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FIELD OF THE INVENTION 

The present invention relates to methods, systems, and programming for display of scaled 
views of web pages. 

BACKGROUND OF THE INVENTION 

This patent application relates to providing scaled views of web pages. This is of particular 
value when using computing devices with small or low resolution screens, such as handheld 
computers, cellphone computers, or computers with wrist or head mounted displays. 

At the time this application is being filed there are multiple handheld computers that have 
approximately 240 by 320 pixel screens that measure approximately four inches diagonally. These 
include the Compaq iPaq H3650 Pocket PC, the Casio Cassiopeia, and the Hewlett-Packard 
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Jornado 525. Unfortunately such a resolution would be too low to display most current Web pages 
on. Currently most Web pages can be viewed with 640 x 480 resolution screen but a significant 
number of web pages are laid out with a minimal width of 800 pixels, and thus are best views at a 
resolution of 800 by 600 or larger. It would be desirable to be able to view most web pages as if 
they were laid out at a resolution of 800 by 600 when they are viewed on hand held devices. 

The manufacturers of liquid crystal displays are now capable of making screens having 
substantially higher resolutions than those that are currently on the market. Makers of organic LED 
displays claim they can achieve even higher resolutions. This means that a four inch diagonal 
screen of the size currently in the handheld computers listed above could have a resolution of 480 
by 640 or higher, and that such resolutions might soon even be on screens as small as 2.5 inches 
in diameter. Although such screens would provide an acceptable resolution for many web sites, 
even a higher apparent resolution would be desirable to view many web pages. 

SUMMARY OF THE INVENTION 

According to one aspect of the invention a method of displaying a web page on a fixed 
pixel resolution screen is provided. The method includes providing a user interface that allows a 
user to select to generate a first set of one or more inputs and to select to generate a second set of 
one or more inputs. The method responds to whether the user has generated the first set of inputs 
or the second set of inputs, respectively, by displaying a given web page on the screen with a first 
scale view or with a second scale view. In the first scale view a column containing text has a first 
width and a text string has a first font size relative the first column width. In the second scale view 
the column has a second, smaller width and the text string has a second font size that is larger 
relative to the second column width than the first font size is relative to the first column width. Each 
character of the text string is displayed each scale views with a font bitmap in which the shape and 
pixel alignment of the character has been selected to improve readability at the pixel resolution at 
which the font bitmap is displayed on the screen in the view. 
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In some embodiments of this aspect of the invention the generation of the first and second 
inputs allows a user to change the display parameters of column width and relative font size as a 
group, without the need to separately select to changes each such display parameter separately. 

In some embodiments in the first scale view the column has a first horizontal displacement 
and in the second scale view the column has a second, smaller horizontal displacement. In these 
embodiments the second font size is larger relative to the second horizontal displacement than the 
first font size is relative to the first horizontal displacement. In some such embodiments the web 
page is displayed with a multi-column layout in which one or more columns are horizontally 
displaced to the right of another column and the horizontal displacement determines the horizontal 
location of such a horizontally displaced column. 

» 

In some embodiments in the first scale view an image has a first size and in the second 
scale view the image has a second, smaller size. In these embodiments the second font size is 
larger relative to the second image size than the first font size is relative to the first image size. 

According to a second aspect of the invention another somewhat similar method of 
displaying a web page on a screen is provided. This method includes providing a user interface 
that allows a user to select to generate a first set of one or more inputs and to select to generate a 
second set of one or more inputs. The method responds to whether the user has generated the 
first set of inputs or the second set of inputs, respectively, by displaying a given web page on the 
screen with a first scale view or with a second scale view. In the first scale view an image has a 
first size; a column, containing text, has a first width; and a text string has a first font size relative 
the first image size and column width. In the second scale view the image has a second, smaller 
size; the column has a second, smaller width; and the text string has a second font size that is 
larger relative to the second image size and column width than the first font size is relative to the 
first image size and column width. The generation of the first and second inputs allows a user to 
change the display parameters of image size, column width, and relative font size as a group, 
without the need to separately select to changes two or more of such display parameters. 
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In some embodiments of this second aspect of the invention in the first scale view the 
column has a first horizontal displacement and in the second scale view the column has a second, 
smaller horizontal displacement and the second font size is larger relative to the second horizontal 
displacement than the first font size is relative to the first horizontal displacement. In such an 
embodiment the generation of the first and second inputs allows a user to change the display 
parameters of image size, column width, column horizontal displacement, and relative font size as 
a group, without the need to separately select to changes two or more of such display parameters. 

In some such embodiments the web page is displayed with a multi-column layout in which 
one or more columns are horizontally displaced to the right of another column and the horizontal 
displacement determines the horizontal location of such a horizontally displaced column. 

Also in some such embodiments in the first scale view the image has a first horizontal 
displacement and in the second scale view the image has a second, smaller horizontal 
displacement and the second font size is larger relative to the second image horizontal 
displacement than the first font size is relative to the first image horizontal displacement. In such 
embodiments the generation of the first and second inputs allows a user to change the display 
parameters of image size, column width, column horizontal displacement, image horizontal 
displacement, and relative font size as a group, without the need to separately select to changes 
two or more of such display parameters. 

In some embodiments of this second aspect of the invention the web page includes a 
specified pixel size for the image and specified pixel width for the column, the second image size is 
a pixel size smaller than the specified pixel size, and the second column width is a pixel width 
smaller than the specified column width. In some such embodiments both the specified pixel size 
of the image and the specified pixel width of the column are scaled down by the same scale factor 
in the second view. In other such embodiments the first image size is a pixel size that is smaller 
than the specified pixel size and larger than the second image size, and the first column width is a 
pixel width that is smaller than the specified column width and larger than the second column 
width. Both the specified pixel size of the image and the specified pixel width of the column can be 
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scaled down by the same first amount in the first view and by the same second, larger amount in 
the second view. 

In some embodiments of this second aspect of the invention the same font size is used for 
the display of the string in the both the first and second views. 

According to third aspect of the invention features of the first and second aspects are 
combined. 

According to a fourth aspect of the invention a method of displaying a web page on a fixed 
pixel resolution screen is provided that includes accessing a web page's contents, including a text 
string and specified pixel width and horizontal pixel displacement relative to the web page for a 
column into which the string is to be laid out. The method lays out and displaying the web page on 
the screen so that: (1) the column is displayed on the screen with a pixel width proportionally 
scaled down relative to the specified pixel width and with a horizontal pixel displacement 
proportionally scaled down relative to the specified horizontal pixel displacement; (2) each 
character of the text string is displayed in the column with a font bitmap in which the shape and 
pixel alignment of the character has been selected to improve readability at the pixel resolution at 
which the bitmap is displayed on the screen; and (3) the font used to display the characters of the 
string represent each character whose shape has two horizontally separated vertical strokes by, in 
at least in one pixel row, having at least one relatively uncovered pixel horizontally placed between 
one or more relatively covered pixels on each side representing the two horizontally separated 
strokes. 

In some embodiments of this fourth aspect of the invention the scaling down of the 
displayed column width and displayed horizontal displacement decreases the pixel size of the 
column width and horizontal displacement by over two times. In some such embodiments the font 
used to display the characters of the string has a majority of lower case characters that have two 
horizontally separated vertical strokes represented by an advance width of 4 pixel columns. Also in 
some such embodiments the scaling down of the displayed column width and displayed horizontal 
displacement decreases the pixel size of the column width and horizontal displacement by at least 
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two and one half times. In some such embodiments the font used to display the characters of the 
string has a majority of lower case characters that have two horizontally separated vertical strokes 
represented by an advance width of 4 pixel columns. 

In some embodiments of this fourth aspect of the invention the web page's contents 
includes an image having a specified pixel size and a specified horizontal pixel displacement and 
the laying out and displaying of the web page on the screen causes the image to be displayed on 
the screen with a pixel size proportionally scaled down relative to the specified pixel size and with a 
horizontal pixel displacement proportionally scaled down relative to the image's specified horizontal 
pixel displacement. 

In some embodiments of this fourth aspect of the invention the web page is displayed with 
a multi-column layout in which one or more columns is horizontally displaced to the right of another 
column and the horizontal displacement determines the horizontal location of such a horizontally 
displaced column. 

The present invention is also relates to apparatuses and computer programming recorded 
in machine readable memory for performing the above methods. 

BRIEF DESCRIPTION OF THE DRAWINGS 

These and other aspects of the present invention will become more evident upon reading 
the , following description of the preferred embodiment in conjunction with the accompanying 
drawings, in which: 

FIG. 1 illustrates a process used according to some aspects of the present invention to 
improve Web browsing and/or display of other types of computer generated content, particularly on 
systems with relatively low-resolution screens. 
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FIG. 2 illustrates a networked computing environment in which the invention can operate 
that includes a portable browser, a proxy server, a Web server, and a font server. 

FIG. 3 illustrates an alternative networked computing environment in which the invention 
can operate that includes a browser and a Web server. 

FIG. 4 illustrates a second alternative networked computing environment in which the 
invention can operate that includes a browser and a Web server. 

FIG. 5 illustrates a third alternative networked computing environment in which the 
invention can operate that includes a browser and a Web server. 

FIG. 6 illustrates a computer system in which the invention can operate that contains 
standard Web content to be displayed and browser functionality containing a process for scaling 
and/or subpixel optimizing that content. 

FIG. 7 illustrates an alternative computer system in which the invention can operate that 
contains the content to be displayed, a proxy process for scaling and/or subpixel optimizing the 
content, and browser functionality. 

FIG. 8 illustrates a second alternative computer system in which the invention can operate 
that contains previously scaled and/or subpixel-optimized content. 

FIG. 9 illustrates a known vertically striped RGB LCD display device. 

FIG. 10 illustrates some of the aspects of the invention involved in performing the subpixel 
optimization of both images and text referred to with regards to steps 108 and 1 12 of FIG. 1, 
respectively. 

FIG. 1 1 illustrates the level of readability provided by one current embodiment of the 
invention when displaying standard Web content on a 320 by 240 color display. 
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FIG. 12 illustrates the mapping of a pixel and subpixel grid over a portion of a higher 
resolution source bitmap image 102. 

FIG. 13 is an expansion of a section of the mapping grid of FIG. 12. 

FIG. 14 illustrates the positioning of a window over the source image used to calculate the 
luminosity of a red (R) subpixel of the lower resolution display device. 

FIG. 15 illustrates the positioning of such a window used to calculate the luminosity of a 
green (G) subpixel of the lower resolution display device. 

FIG. 16 illustrates the positioning of such a window used to calculate the luminosity of a 
blue (B) subpixel of the lower resolution display device. 

FIG. 17 illustrates scan lines used in a scan line coverage method to calculate the subpixel 
luminosity of a red subpixel on a display device having a relatively low resolution by estimating the 
portion of the red subpixel's associated window in a higher resolution source image that is covered 
by one or more pixel of different colors. 

FIG. 18 illustrates similar scan lines used to calculate the luminosity of a green (G) 
subpixel of the lower resolution display device shown in FIG. 17. 

FIG. 19 illustrates similar scan lines used to calculate the luminosity of a blue (B) subpixel 
of the lower resolution display device shown in FIGS. 17 and 18. 

FIG. 20 is a repeat of FIG. 17 provided on the same sheet as FIGS. 21 and 22 for ease of 
comparison. 

FIG. 21 illustrates the portions of the horizontal scan line shown in FIG. 20 that are 
covered by different source image pixels within the red pixel's source image window. 
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FIG. 22 illustrates the portions of the vertical scan line shown in FIG. 20 that are covered 
by different source image pixels within the red pixel's source image window. 

FIG. 23 is identical to FIG. 18 and is provided on the same sheet as FIGS. 24 and 25 for 
ease of comparison. 

FIG. 24 illustrates the portions of the horizontal scan line shown in FIG. 23 that are 
covered by different source image pixels within the green pixel's source image window. 

FIG. 25 illustrates the portions of the vertical scan line shown in FIG. 23 that are covered 
by different source image pixels within the green pixel's source image window. 

FIG. 26 is identical to FIG. 19 and is provided on the same sheet as FIGS. 27 and 28 for 
ease of comparison. 

FIG. 27 illustrates the portions of the horizontal scan line shown in FIG. 26 that are 
covered by different source image pixels within the blue pixel's source image window. 

FIG. 28 illustrates the portions of the vertical scan line shown in FIG. 26 that are covered 
by different source image pixels within the blue pixel's source image window. 

FIG. 29 is a highly simplified pseudocode description of a subpixel optimization method 
that calculates subpixel luminance values based on line coverage values, such as the line 
coverage values illustrated with regard to FIGS. 17 through 28. 

FIG. 30 illustrates how two horizontal and two vertical scan lines can be used on alternate 
embodiments of "line coverage" methods for calculating the colors of pixels in subpixel-optimized 
scaled images. 
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FIG. 31 illustrates how two diagonal scan lines can be used on alternate embodiments of 
"line coverage" methods for calculating the colors of pixels in subpixel-optimized scaled images. 

FIG. 32 illustrates how a combination of two diagonal, one horizontal, and one vertical 
scan lines can be used on alternate embodiments of line coverage" methods for calculating the 
colors of pixels in subpixel-optimized scaled images. 

FIG. 33 illustrates line coverage for two horizontal coverage lines at a 1/2 horizontal and 
vertical scaling. 

FIG. 34 illustrates line coverage for two vertical coverage lines at the 1/2 horizontal and 
vertical scaling shown in FIG. 33. 

FIG. 35 illustrates line coverage for two horizontal coverage lines at approximately a 2/5 
horizontal and vertical scaling. 

FIG. 36 illustrates line coverage for two vertical coverage lines at approximately the 2/5 
horizontal and vertical scaling shown in FIG. 35. 

FIG. 37 illustrates line coverage for two horizontal coverage lines at approximately a 2/3 
horizontal and vertical scaling. 

FIG. 38 illustrates line coverage for two vertical coverage lines at approximately the 2/3 
horizontal and vertical scaling shown in FIG. 37. 

FIG. 39 illustrates the source image pixel window used in an "area coverage" method of 
calculating the color values of a subpixel-optimized scaled image. 

FIG. 40 is similar to FIG. 39 except that it uses different hatching to illustrate the areas of 
different source image pixels within a source image window used to calculate a subpixel's 
luminosity value according to one such "area coverage" method. 
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FIG. 41 is a highly simplified pseudocode description of a subpixel optimization method 
that calculates subpixel luminance values based on an area coverage values, such as those 
discussed with regard to FIGS 39 and 40. 

FIG. 42 illustrates a source image window and associated scan lines that can be used in 
the production of a scaled bi-color subpixel-optimized image of a bitmap image to associated a 
luminosity value with a red subpixel. 

FIG. 43 illustrates a source image window and associated scan lines that can be used in 
the production of a scaled bi-color subpixel-optimized image of a bitmap image to associate a 
luminosity value with a green subpixel. 

FIG. 44 illustrates a source image window and associated scan lines that can be used in 
the production of a scaled bi-color subpixel-optimized image of a bitmap image to associate a 
luminosity value with a blue subpixel. 

FIG. 45 illustrates the luminosity of a set of source image gray scale pixels associated 
with a portion of an RGB subpixel display grid. 

FIG. 46 illustrates how the luminosity associated with an individual subpixel shown in FIG. 
45 is distribution under a traditional linear filtering method. 

FIG. 47 illustrates subpixel luminosity values that result from the traditional linear filtering 
method shown in FIG. 46 being applied to multiple subpixels in an pixel row. 

FIG. 48 illustrates the distribution of the minimum subpixel luminosity values under a non- 
linear filtering. 

FIG. 49 illustrates the distribution of the excess luminosity values under a continuation of 
the non-linear filtering method shown in FIG. 48. 
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FIGS. 50 through 52 compare the results of the linear and non-linear filtering methods with 
the original source pixel luminosities. 

FIG. 53 is a highly simplified pseudocode representation of a software method for creating 
a subpixel-optimized representation of a bicolor bitmap. 

FIG. 54 is a flow diagram of a process to allow a user to dynamically tradeoff color and 
positional resolution. 

FIG. 55 illustrates the mapping between a character-font shape defined by an outline font 
description and an array of pixels having subpixels used to represent that shape on a subpixel 
addressable display; 

FIG. 56 is a screen shot of a 320 x 240 screen of a web page produced by one 
embodiment of the present invention; 

FIG. 57 is a 2x blowup of the screen shot shown in FIG. 56; 

FIG. 58 illustrates how a computer can access font bitmaps or font outlines from a font 

server; 

FIG. 59 illustrates how a computer can access font bitmaps or font outlines that it has 
stored within it; 

FIG. 60 is a highly simplified pseudocode representation of an algorithm for calculating a 
subpixel-optimized bitmap of a character-font shape using non-linear color balancing of the type 
illustrated in FIGS. 48 through 52; 
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FIGS. 61 through 63 illustrate the size of the source image window in a character-font 
shape image, such as that shown in FIG. 55, used to determine a coverage value for each of the 
three subpixel's of an individual pixel of a pixel array such as that shown in FIG. 55; 

FIGS. 64 through 67 illustrates some prior art techniques that have been used to calculate 
coverage values for non-square rasterization units (usually whole pixels in the prior art); 

FIGS. 68 through 87 illustrate a computationally efficient method of calculating the 
coverage value of rasterization units, using weighted line coverage values, which method is used in 
some embodiments of the present invention to calculate a coverage value for subpixels; 

FIGS. 88 through 90 illustrate some of the other arrangements of coverage lines that can 
be used with a weighted line coverage algorithm of the general type described with regard FIG. 68 
through 87; 

FIG. 91 illustrates a mapping of an array of pixels, and their respective subpixels, into an 
image of a portion of a hypothetical font outline; 

FIG. 92 illustrates corresponding coverage values that have been calculated for the 
subpixel's of the array shown in FIG. 91 ; 

FIG. 93 corresponds to FIG. 46, and like FIG. 46 illustrates how a prior art linear color 
balancing method distributes all of an individual subpixel's coverage value over a series of adjacent 
subpixels within a given pixel row; 

FIGS. 94 and 95 illustrates color balance filters that can be used with the non-linear color 
balancing method described with regard FIG. 60; 

FIG. 96 is a highly simplified pseudo code description of an algorithm that can be used to 
represent the whole pixel alpha values calculated for font bitmaps by a method such as that shown 
in FIG. 60 into a more limited Color Space of such whole pixel alpha values; 
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FIG. 97 is a highly simplified pseudo code description of an algorithm for displaying text 
strings on a subpixel addressable display using font bitmaps created by a combination of the 
methods illustrated in FIG. 60 and 96; 

FIGS. 98 through 101 illustrate how well the present invention can display web pages on a 
320 x 240 screen, with FIGS. 98 and 100 each being a screen shot of a 640 x 480 layout of a 
different web page, and FIGS. 99 and 101 showing how the present invention is capable of 
displaying each of these two web pages on a 320 x 240 screen; 

FIG. 102 is a schematic block diagram of some of the data structures and programming 
used by a proxy server and thin client computer to enable a user of the thin client computer to 
access web content on a scaled-down, subpixel-optimized screen; 

FIG. 103 is a portion of the HTML code of the web page illustrated in FIGS. 98 and 99; 

FIG. 104 illustrates the layout of a web page produced by the proxy server and the portion 
of that layout that falls within the proxy server's virtual screen, which in this example corresponds 
to the portion of the web page shown in FIG. 99; 

FIGS. 105A and 105B are highly simplified pseudo code descriptions of programming on 
the proxy server shown in FIG. 102; 

FIGS. 106A through 106C are highly simplified pseudo code descriptions of proxy server 
programming for capturing, scaling-down, and subpixel-optimizing a representation of a portion of a 
web page and downloading it to a thin client computer; 

FIG. 107 is a highly simplified pseudocode description of programming for the actual 
downloading of the web page representation captured by the programming of FIGS. 106A through 
106C; 
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FIG. 108 is a highly simplified representation of the data downloaded to a thin client 
computer by the programming illustrated in FIG. 107; 

FIGS. 109A through 109C are highly simplified pseudocode representations of 
programming on the thin client shown in FIG. 102; 

FIGS. 1 10 through 112 illustrate how, if a user clicks on a text entry field on a web page's 
display on the thin client shown in FIG. 102, a pop-up keyboard is shown that allows the user to 
enter text into that field; 

FIG. 113 illustrates how the user can use the same pop-up keyboard to enter URLs that 
allow him or her to access desired websites; 

FIG. 1 14 is virtually identical to FIG. 13 except that it illustrates an embodiment of the 
invention having a toolbar at the top of the thin client computer screen that includes graphical user 
interface buttons and a URL text entry field; 

FIGS. 115 and 116 are highly simplified pseudocode representations of programming on a 
proxy browser and thin client computer, respectively, that is used in an alternate embodiment of the 
invention in which the thin client computer stores the layout of an entire web page to allow it to 
more quickly scroll and zoom relative to that web page; 

FIG. 1 17 is a schematic illustration used to help explain the operation of the pseudocode 
illustrated in FIGS. 115 and 116; 

FIGS. 1 18 through 120 illustrate how the present invention can allow a user to rapidly 
select a portion of a web page or a screen and then zoom to that selected portion; 

FIGS. 121 through 128 illustrate a zoomclick aspect of the invention that allows a user to 
view and select portions of a web page or a screen with greater accuracy; 
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FIG. 129 is a highly simplified pseudocode description of programming for allowing a user 
to have selected text re-flowed across a given screen width at a larger scale; 

FIGS. 130 through 137 are used to illustrate how the text re-flow capability shown in FIG. 
129 can operate; 

FIG. 138 is used to illustrates how multiple client computers can be programmed to access 
a common font server and/or a common proxy server; 

FIG. 139 is a highly simplified pseudocode description of programming that can be used 
on one or more font servers; 

FIG. 140 illustrates how certain aspects of the present invention can be used to allow a 
client computer to view screens that are drawn by applications (which can include, but are not 
limited to, one or more web browsers) running on a remote computer in a scaled-down, subpixel- 
optimized manner by intercepting calls made by such applications to the remote computer's 
operating system; 

FIG. 141 illustrates how subpixel-optimized, scaled-down views can be had of screen 
output generated by application programs (which can include but are not limited to one or more 
web browsers) running on a given computer, even if those applications have not been programmed 
to generate such views, on the screen of that given computer, by intercepting calls to the 
computer's operating system made by such applications; 

FIG. 142 illustrates how certain aspects of the present invention can be used allow 
portable small-screen, thin-client computers to access web content and the screen output of 
various application programs through both local and/or Internet wireless communication; 

FIGS. 143 and 144 are used to illustrate how in some embodiments of the present 
invention subpixel-optimized output is displayed with a landscape orientation by rotating a 
computing device that has an operating system programmed to work in a portrait orientation; 
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FIG. 145 is a highly simplified pseudocode description of programming use to draw a 
simple shape with a subpixel-optimized resolution; 

FIG. 146 is a highly simplified pseudocode description of how web applets can be used to 
draw subpixel-optimized elements on the screen of a computer; 

FIG. 147 is a highly simplified block diagram illustrating how rollover images can be 
subpixel-optimized; 

FIG. 148 is a highly simplified block diagram illustrating how GIFF animations can be 
subpixel-optimized; 

FIG. 149 is a highly simplified pseudocode description of how 3-D animation can be 
subpixel-optimized; 

FIGS. 150 and 151 are highly simplified pseudo code descriptions of how a client/server 
gaming system can be used to provide subpixel-optimized game images on a client computer; 

FIG. 152 is a highly simplified pseudocode description of how subpixel-optimized displays 
can be made of images having corresponding transparency maps; 

Figure 153 is a highly simplified pseudocode description of how video using interpolation 
between keyframes can be subpixel-optimized; 

FIG. 154 is a highly simplified pseudocode description of how video whose representation 
includes the drawing of screen changes to less than a whole frames can be subpixel-optimized; 

FIGS. 155 and 156 are highly simplified pseudocode description of different methods of 
displaying images that move relative to a display window; 
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FIGS. 157 through 159 are highly simplified pseudocode description of how subpixel 
optimization can be applied to video that is been represented by various compression techniques; 

FIG. 160 is a highly simplified pseudocode representation of programming for enabling a 
server computer to download subpixel-optimized, scaled-down video to a client computer; 

FIG. 161 is a highly simplified pseudocode description of programming on both a client and 
proxy computer to enable the client computer to access scaled-down, subpixel-optimized video 
from other servers through a proxy computer; 

FIG. 162 is a highly simplified pseudocode representation of programming that allows 
electronic ink to be viewed more clearly; 

FIGS. 163 through 166 are used to help illustrate the benefits of the programming describe 
with regard FIG. 162; 

FIG. 167 illustrates that the present invention relates not only to methods, but also to 
programming and data related to such methods stored in a machine readable form or embodied in 
a propagated signal, and to programmed and/or hardwired computer systems for performing such 
methods and/or use such programming and/or data. 

FIGS. 168 through 184 are used to describe additional improvements to the invention for 
improving the clarity of color-balanced subpixel-optimized font bitmaps produced by the present 
invention. 

FIG. 1 85 is a higher level description of the selected-text reflow method described with 
regard to FIGS. 129 through 134; 

FIG. 186 is a high-level pseudocode description of a zoom-to-fit method;, of the general 
type described with regard to FIGS. 1 18 through 120; 
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FIG. 187 is a high-level pseudocode description of a drag scroll method, that allows a user 
to easily navigate within the display of a web page's layout; 

FIG. 188 is a high-level pseudocode description of a click-zoom method that enables the 
user to rapidly selected to zoom in on a desired portion of the display of a layout of a web page; 

FIG. 189 is a highly simplified pseudocode description of the zoomclick method described 
with regard to FIGS. 121 through 128; 

FIG. 190 is a highly simplified pseudocode description of a method that allows a user to 
see a zoom-out view of a web page using greeking; 

FIG. 191 is a highly simplified pseudocode description of programming on a client 
computer, such as that shown in FIG. 102, that cause different scale views of a web page to be 
shown on that client computer; 

FIG. 192 is a highly simplified pseudocode description of alterations to the proxy server 
programming shown in FIG. 105B that support the pseudocode shown in FIG. 191; 

FIG. 193 illustrates a client computer, such as that shown in FIG. 102, displaying a web 
page at a virtual layout size of 640 by 480; 

FIG. 194 shows the screen of FIG. 193 after a user has selected to display a pop-up 

menu; 

FIG. 195 show the screen of FIG. 194 after the user has selected to display a view size 
submenu; 

FIGS. 196 through 198 show the same web page as is shown in FIGS. 193 through 195 
displayed, respectively, at the 480 by 320, 640 by 480, and 800 by 600 virtual resolution; 
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FIGS. 199 and 200 show the same web page displayed, respectively, on the 320 by 240 
screen shown in FIG. 193 using the 800 by 600 virtual resolution and on an actual 800 by 600 
display using a normal browser; 

FIGS. 201 through 203 show another web page displayed, respectively, on the 320 by 240 
screen of FIG. 193 using the 640 by 480 virtual resolution; on the same 320 by 240 screen using 
the 800 by 600 virtual resolution; and on an actual 800 by 600 screen using a normal browser; 

FIG. 204 is a 4X blow-up of a portion of the font bitmaps shown on the 800 by 600 virtual 
resolution view shown in FIG. 202 near the center of that figure; 

FIG. 205 is a 4X blow-up of the font bitmaps shown FIG. 204 encircled in dotted lines; 

FIGS. 206 through 209 illustrate portions of font bitmaps shown in FIG. 205 in which two 
vertical strokes appear in the same horizontal portion of a character's shape separated by a 
relatively uncovered pixel. 

DETAILED DESCRIPTION OF SOME PREFERRED EMBODIMENTS 

FIG. 1 is a high level diagram that represents basic processes and data representations 
that may be used according to some aspects of the present invention to improve Web browsing 
and/or display of other types of computer generated content, particularly on systems with low 
resolution displays. 

Digital content 100, including one or more bitmap images 102 and text 104 shown at the 
top of FIG. 1 is displayed in a subpixel-optimized downscaled format 106 shown at the bottom of 
that figure. In one embodiment of the invention a difference process, comprised of step 108 is 
used to subpixel-optimize the display of the bitmap images 102, than is used to subpixel optimize 
the display of the text content. Step 108 uses a subpixel optimization routine that is particularly 
suited for producing subpixel-optimized images from color bitmaps. The process 108 also scales 
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down the bitmaps for display on screens having a lower resolution than that at which most Web 
content is currently displayed. 

The text 104 contained in the digital content 100 is processed for display on a small- 
resolution subpixel-addressable screen by using steps 1 10 and 1 12. Step 110 replaces the fonts 
normally used to display text with fonts that are optimized for display at small resolutions on 
subpixel-optimized screens. Then step 112 uses font bitmaps from the substituted fonts that have 
been produced by a subpixel optimization routine particularly suited for the representation of high 
resolution images of shapes of a uniform color, such as the mathematically defined outlines 
commonly used to define font shapes. 

One use of the present invention is in the context of a portable, low resolution Web 
browser that displays images and/or text, represented by a markup language, that have been 
downloaded from the Internet. 

To date there have been multiple so-called mark-up languages. One of the earliest and 
most successful was SGML (Standard General Markup Language). SGML is a text-based 
language that can be used to 'markup' data with descriptive 'metadata' that provides information 
about the data. As an example, markup metadata can be used to indicate the purpose for which 
the data is intended or the location within a document's visual presentation at which the data 
should be positioned. It can also be used to indicate links to data of other types, such as images, 
which are to be inserted at a given location in a text, or in a document described by the mark-up 
language. Several markup languages that are commonly used today, such as HTML and XML, are 
derived from SGML. 

In a preferred embodiment of the present invention, the digital content 100 referred to in 
FIG. 1 above may be standard Web content that includes text and/or images represented by a 
markup language such as HTML. This standard Web content 100, representing perhaps a Web 
site home page, can be downloaded through various apparatus and methods described below for 
display on a portable low resolution browser device 200, shown in FIGS. 2 through 4. Before 
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display on the browser device 200, the digital content 100 may be scaled and/or subpixel-optimized 
for enhanced readability through various methods and processes, such as those described below. 

FIG. 2 illustrates a networked computer environment implemented in accordance with one 
embodiment of the present invention. The thin client browser 200 program runs in a handheld or 
other small computing device capable of retrieving and displaying text and/or graphics on a small 
display screen, such as, for example, a liquid crystal display (LCD) screen. The browser allows a 
user to request digital information from a remote source, e.g., from the Internet, and to display it on 
a screen. 

In the embodiment of the present invention illustrated in FIG. 2, a user would request the 
retrieval and display of digital content, containing images and/or text, by way of manipulation of the 
controls of the thin client browser 200. The requested digital content may be a specific Web page 
accessible over the Internet. The thin client browser 200 then makes its request 202 for content 
through a physically remote proxy server 210 over a network 138, which can be, for example, a 
LAN, a WAN, or the Internet. 

The proxy server 210 runs a proxy process 216 that responds to the request for digital 
content by generating a corresponding request 214 to a physically remote Web server 220 that 
contains the digital content 100 requested by the user. Server 220 responds to the proxy server 
request 214 by a download 222 of the digital content 100 over the network 138 to the proxy server 
210. 

The proxy process 216 within the proxy server 210 then uses its computational resources 
to scale and subpixel optimize the digital content 100, including performing the functions 108 and 
1 1 0 illustrated in FIG. 1 . Scaling and subpixel optimizing are aspects of the present invention that 
result in the enhanced readability of images, such as text and/or graphics, on small display 
devices. They will be discussed in greater detail in a subsequent section. 
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The proxy server 210 completes a download 212 of the now scaled and subpixel-optimized 
content to the browser 200. At this point, the user is able to view the content on the screen of the 
browser 200. 

In the embodiment of the invention shown in FIG. 2, the text portion of the digital content is 
downloaded to the browser in the form of one or more strings of characters and associated 
designations of the font family, font size, and other font attribute. The thin client browser performs 
the function 112 shown in FIG. 1 by displaying the strings with images composed from individual 
subpixel-optimized font bitmaps. If the thin client does not have bitmaps for any character in such 
a string in the font size and family specified for it, it requests one or more such bitmaps from a font 
server 230. In various embodiments of the inventions shown in FIGS. 2 through 8, the proxy 
server could provide such font bitmaps or the thin client could have them as a standard part of its 
software (although that would increase the size of the browser software). In still other 
embodiments the fonts could be outline fonts. One advantage of font bitmaps is that some font 
vendors are more willing to allow bitmaps of their fonts to be distributed more freely than outlines of 
such fonts. 

An alternate embodiment of the present invention is illustrated in FIG. 3. In this 
embodiment the proxy server 210 and the Web server 220 of FIG. 2 are replaced with a single 
remote server 220A. The thin client browser 200 makes its request 202A for digital content 100 to 
the remote server 220A over a network 138. For example, network 138 may be the Internet or a 
LAN and the digital content 100 may be a specific Web page. The remote server 220A contains 
the requested digital content 100 and runs a proxy process 216A that responds to the request 
202A. This proxy process 21 6A can be any process running on the server that dynamically scales 
and/or subpixel optimizes web content for display on the thin client browsers. The proxy process 
21 6A operates upon the stored digital content 100 and dynamically converts it to the form 106 
shown in FIG. 1 by performing the steps 108 and 110 of FIG. 1. The remote server 220A 
completes a download 21 2 of the scaled and/or subpixel-optimized content to the thin client 
browser 200. 
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Another alternate embodiment of the invention is illustrated in FIG. 4. As in FIG. 3 the thin 
client's request is made directly to a remote server, in this case server 220B, over a network 138. 
In this implementation the remote server 220B contains the requested digital content in both a 
standard form 100, that is for use by standard browsers computers, and the scaled and/or 
subpixel-optimized content 100A. The conversion from standard digital content 100 to the scaled 
and/or subpixel-optimized form has occurred in advance, thereby eliminating the need for a proxy 
process to dynamically convert it. The thin client provides information to the server indicating that 
it should receive the scaled and/or subpixel-optimized version of the requested content. The 
remote server 220B completes a download 212 of the scaled and/or subpixel-optimized content 
100 to the thin client browser 200. 

A further alternate embodiment is illustrated in FIG. 5. Browser 200A is a full scale 
browser that also contains a scaling and/or subpixel optimization process 510. The browser 200A 
makes a request 202B to a remote server 220C over the network 138 for the digital content 100. 
Server 220C completes a download 21 2A of the requested digital content 100 to the browser 200A. 
The conversion of the digital content 100 to a scaled and/or subpixel-optimized form is handled by 
the process 510 running in the browser 200A. 

FIG. 6 illustrates a single computer system 600 capable of scaling and/or subpixel 
optimizing digital content 100. In this preferred embodiment the digital content has been created 
on or loaded into computer system 600 in advance. Computer system 600 contains a browser 
process 620 that includes a scaling and/or subpixel optimizing sub-process 640. Here the user 
makes a request to the computer system 600 by way of an attached input device, e.g., a keyboard 
or mouse, for the display of the digital content 100. The browser process 620 retrieves the 
requested digital content 100 from one of the computer system's storage elements, e.g., electronic 
memory or disk storage. Once retrieved, the browser process 620 then passes the digital content 
to the scaling and/or subpixel optimizing sub-process 640. Once the conversion is complete, the 
converted content is displayed on the display screen of the computer system 600. This 
embodiment of the present invention operates without the need for a network or remote servers. 
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FIG. 7 illustrates an alternate single computer system implementation. In this 
embodiment, computer system 700 contains the digital content 100 (e.g., the contents of a specific 
Web^page) that has been created or loaded in advance, a proxy process 740, and a browser 
process 720. The proxy process 740 executes scaling and/or subpixel optimization programming 
760. The browser process passes a user request for display of the digital content 100 to the proxy 
process 740. Proxy process 740 then retrieves the digital content 100 from the storage element of 
computer system 700. Once retrieved, the programming 760 converts the digital content 100 to a 
scaled and/or subpixel-optimized form that is then passed to the browser process 740 for display 
by the display device of computer system 700. 

FIG. 8 illustrates a second alternate single computer system implementation. Here 
computer system 800 contains scaled and/or subpixel-optimized Web content 810. A browser 
process 820 handles user requests for display of the content 100A, retrieves it from a storage 
element of computer system 800, and displays it on the screen of computer system 800. 

In the embodiments of the present invention presented above, the scaling of the image 
from the resolution of the source image to the resolution of the subpixel addressed screen has 
been determined, in part, by the respective resolutions of the source image and the subpixel 
addressed display screen. 

In some embodiments of the invention, the determination of the scaling factor between the 
source image resolution and the resolution to be displayed on the subpixel addressable display 
screen can be specified by the user of the browser device. In this embodiment the user of the 
browser selects from a plurality of scale factors by communicating the scale factor to the process 
that scales down an image read from storage. The process that scales down the image read from 
storage then scales down and subpixel optimizes the image by a horizontal and vertical scale 
factor that varies as a function of the selected scale factor. 

As with most other user inputs to the browser device, such scale selections can be made 
by use of physical or GUI buttons, menu items, dialog boxes, or any other known user interface 
device on the browser device. 
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In some such embodiments the user of the browser device may choose a second scaling 
factor from a plurality of scaling factors, according to which the digital content will be re-scaled and 
re-subpixel optimized and redisplayed, after the image has been previously retrieved from storage 
and displayed in subpixel-optimized form at a first scaling factor. 

In such embodiments, the scaling factor used in the first scaled and subpixel-optimized 
display may have been as a result of a default or preferred scaling factor or it may have been as a 
result of a scaling factor previously chosen by the user of the browser device. The user of the 
browser device may choose from a plurality of scaling factors for the redisplay of the digital content 
by the method of manipulating an input apparatus of the browser device. Such manipulation of the 
input apparatus of the browser device will cause the image to be scaled according to the second 
chosen scaling factor. 

Such a second scaling may occur as a result of a process running either within the 
browser device or within a physically remote server. The user of the browser device may continue 
to select from a plurality of scaling factors for subsequent redisplays. 

It is easiest to downscale digital images by integer multiples, which cause an integer 
number of pixels in a source image to fit into a given pixel in the resulting downscaled image. For 
example, the scaling from a 640 by 480 resolution to a 320 by 240 resolution is a downscaling by a 
factor of two. Some embodiments of the present invention allow the user to select from a plurality 
of downscale factors, including non-integer downscaling factors. An example of a non-integer 
downscaling factor is that of a 3/2 downscaling factor that would cause a 480 by 360 pixel portion 
of a 640 by 480 resolution source image to be scaled and/or subpixel optimized for display on a 
320 by 240 resolution display screen. 

In this specification when we refer to the resolution of a web page, we normally mean the 
resolution at which a browser program doing the layout has been requested to lay out the web 
page's content. As is described below, some web page's have elements that require a certain 
minimum layout width, and if a request is made to display the web page at less than such a width, 
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those elements will be laid out at their minimum width even if that width is larger than the requested 
layout width. 

Computer graphic displays such as cathode ray tubes (CRT) or liquid crystal display (LCD) 
screens almost exclusively use the RGB model of color space, although the invention can be used 
with other color models, such as the CMYK color model. In the RGB model, the three primary 
additive colors, red, green, and blue, are blended to form a desired color as perceived by the 
human eye. 

Most portable computing or imaging devices have LCD screens that use the RGB model. 
Such LCD screens are comprised of a rectangular array of thousands of grid elements, referred to 
as pixels, each capable of displaying any one from a large number of color values from an RGB 
color space, that when perceived as a whole, form an image. LCD screens are characterized by 
the number of horizontal and vertical pixels they contain. 

Each pixel in turn is composed of three individually addressable sub-components, referred 
to here as subpixels. Most commonly, the three subpixels are rectangular red, green, and blue 
elements. In the most common implementation, the three red, green, and blue subpixels are each 
assigned a luminous intensity value such that they blend together to give the entire pixel the 
appearance of the desired color. All of the pixels on an LCD screen blend together, in turn, to give 
the appearance of the desired image. 

The subpixels are considered individually addressable because the color value assigned to 
an individual pixel has a separate red, green, and blue color component, or luminosity value, which 
will be displayed, respectively, by the red, green, and blue subpixels of that pixel. Thus, the 
luminosity of each subpixel can be separately controlled by controlling the value of its associated 
color component's luminosity value in the color value assigned to the pixel. 

In an LCD device and other "subpixel addressed" displays, such as color LED (including 
screens using organic light-emitting diodes (OLEDs)) or gas plasma displays, each individual 
subpixel has a fixed, known position on the display. Many display devices, such as almost all 
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cathode ray tube (CRT) displays are not subpixel addressable. For example, although each pixel 
of a CRT has an individual luminosity value for each of its red, green, and blue component colors, 
the exact physical location within each such pixel of the elements that generate the light associated 
with those different color values is normally not known because it varies as a function of the 
individual phosphor pattern of the screen, the resolution of the horizontal and vertical scan, and the 
current exact state of the voltages that control the exact locations at which individual pixels are 
drawn on the screen. 

FIG. 9 illustrates a 12x12 portion of an LCD screen 900 that is comprised of a plurality of 
pixel rows (R1-R12) and pixel columns (C1-C12). Each intersection of a row and a column 
constitutes a pixel element. Actual implementations of LCD screens can have an arbitrary number 
of rows and columns, though grids of 320 by 240, 640 by 480, 800 by 600, 1024 by 768, and 1280 
by 1024 are frequently seen. 

Pixel R1-C1 is shown within a dotted circle 910. Pixel R1-C1 is itself made up of three 
pixel sub-components herein referred to as subpixel elements. An expanded view of pixel C1-R1 
is shown as expanded pixel 920 at the bottom of FIG. 9. Subpixel element 902 displays as red, 
subpixel element 904 displays as green, and subpixel element 906 displays as blue. The individual 
subpixel elements 902, 904, and 906 are approximately 1/3 of the width of a full pixel and are equal 
in height to a full pixel. 

As illustrated in LCD screen 900, when a plurality of such pixels are arrayed in a grid this 
causes the appearance of vertical color stripes down the LCD screen 900. This known 
arrangement of pixels is sometimes referred to as vertical RGB striping. Other known 
arrangements lay the pixel elements out in the orthogonal direction such that horizontal striping 
results (in which case rotating the screen by 90 degrees will convert it into a vertically striped 
screen). 

In common usage, the luminous intensity of the three subpixel elements of a pixel are set 
such that the pixel is perceived by the human eye as being of the desired hue, saturation, and 
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intensity. The RGB subpixel elements are used together to form a single colored pixel to represent 
a single sample of an image to be displayed. 

One aspect of the present invention relates to the improvement of the readability of 
downloaded Web content, and other digital content including text and images, on low-resolution 
screens, such as, for example, displays having column by row pixel ratios of 320 by 240 or 240 by 
320 (in which case they can be rotated 90 degrees to have a resolution of 320 by 240). Many of 
the embodiments of the present invention discussed and shown in some detail map image and text 
from a virtual layout resolution of 640 by 480 pixels onto a screen with a 320 by 240 pixel 
resolution. But the present invention can be used with other resolution screens. To give just a few 
examples, it could be used to display content laid out roughly as it would look at a 1024x768 
resolution on a 512x384 resolution screen, or display content laid out roughly as it would look at a 
800x600 pixels on a 400x300 screen. In other embodiments, the invention can be used with 
relatively low-resolution displays that have pixel dimensions that are other than even fractions of 
the horizontal and/or vertical pixel .dimensions common on personal computer screens. 

In general when we refer to a small resolution screen we mean a screen having a smaller 
resolution than that on which given digital content or a given layout of digital content would 
normally be intended to be displayed upon. By such smaller screens we also mean to include 
portions of larger screens, such as windows on larger screens, that have such lower resolution. 

In FIG. 10, image content 105 and text content 107 represent a portion of the subpixel- 
optimized display 106 of FIG. 1. The image shown in FIG. 1 is a grayscale blowup of the actual 
color values associated with the subpixel-optimized display of both text and images. The portion of 
the image content 105 contained within the rectangle 1000 of FIG. 10 is shown expanded at 1020 
to make its individual pixels easier to see. Correspondingly, a portion of the text content 107 
contained within rectangle 1040 is shown expanded at 1060. 

It is important to note that the pixels shown at 1020 and 1060 represent whole pixels 
because the software used to generate the images 1020 and 1060 merely represents the 
grayscale corresponding to the RGB color values associated with individual whole pixels. The 
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subpixel blowups 1020A and 1060A are attempts to represent the intensity of each of the three 
subpixels associated with each pixel in the blowups 1020 and 1040, respectively. 1020B is a 
blowup corresponding in scale and location to blowups 1020A and 1020. In it the pixel grid of the 
image is displayed in relatively bold lines, and the three subpixel divisions within each such pixel 
are shown in somewhat finer lines. This composite grid is superimposed on top of the original 
higher resolution color bitmap image 102 of FIG. 1 from which the pixilation patterns shown in the 
blowups 1020 and 1020A have been derived. In the particular images shown, the resolution of the 
color bitmap 102 is twice as high in both the vertical and horizontal direction as the whole pixel 
resolution in the image 105 shown at the bottom of FIG. 10. 

The blowup 1060B illustrates how a subpixel-optimized image of a font is produced by 
determining the portion of the area of each subpixel that is covered by a high-resolution font outline 
of the one or more characters involved. 

As can be seen by comparing the subpixel resolution blowups 1020A and 1060A to the 
corresponding whole pixel blowups 1020 and 1060, respectively, the display of subpixel-optimized 
representations of images and text at subpixel resolution provides better spatial resolution. 

FIG. 1 1 provides a representation of the quality of the readability provided by an 
embodiment of the invention when displaying standard Web content on a 320 by 240 color display. 
Bitmap 1 100 at the top of the figure is a grayscale, whole-pixel blowup of an actual bitmap 
produced from a standard 640 by 480 layout of a portion of a priceline.com web page. This image 
corresponds to the portion of the web page contained within the rectangle 1 130 shown in the whole 
screen 320 by 240 image of the web page at the bottom of FIG. 1 1 . Bitmap 1 120 in the middle of 
the figure is a grayscale, whole-pixel blowup of the color bitmap of the same portion of that 320 by 
240 image. 200B at the bottom of FIG. 1 1 represents a hand-held computing device that is 
functioning as a thin client browser of the type described with regard to FIG. 2. On the screen of 
this browser is shown a representation of a 320 by 240 subpixel-optimized bitmap representing a 
640 by 480 layout of the web page. Like the blowup 1020 of FIG. 10, the bitmap 1 130 shown at 
the bottom of FIG. 11 illustrates individual pixels with grayscale levels corresponding to the 
average luminosity of whole pixels. When this image is seen on a 320x240 screen having vertical 
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subpixel striping in the landscape orientation, the actual image appears to have an even higher 
resolution, as indicated by the blowups 1020A and 1060A in FIG. 10. 

Any known algorithm for deriving subpixel-optimized images of color bitmaps can be used 
for the purposes of many aspects of the present invention. In one embodiment of the present 
invention, the luminosity assigned to each given subpixel of a given color is determined by the 
average intensity of that given color's value in each pixel of the source image that is totally or 
partially inside a rectangular window in the source image associated with the given subpixel. This 
source image window has a size and position relative to the source image that corresponds to the 
area of a whole pixel in the scaled down image that is centered around the location of the given 
subpixel in that scaled down image. The average intensity assigned to the subpixel is calculated by 
multiplying the intensity of each source image pixel that totally or partially covers the source image 
window by the percent of that window's area covered by each such source image pixel. 

FIG. 12 illustrates the mapping of a higher resolution source image 102 into a reduced 
resolution display's subpixel grid. This figure illustrates the subpixel grid 1210 being superimposed 
on a portion of the original higher resolution color bitmap 102 shown in FIG. 1. Circle 1220 
encloses an area of that grid that corresponds to one pixel in the intended lower resolution display 
device. The position and scale of the grid patterns is determined by the relationship between the 
higher resolution source bitmap image and the pixel grid of the resulting subpixel-optimized image. 
The particular grid pattern 1210 shown in FIG. 12 represents a scaling from the pixel resolution of 
the color bitmap image 102 to a display screen resolution that has one-half as many pixels in both 
the horizontal and vertical direction as the source image. An example of this scaling is that of an 
image having pixelation appropriate for display on a 640 by 480 display being scaled down for 
proportional display on a 320 by 240 display screen. Thus, each bold line division of grid pattern 
1210 covers four pixels of color bitmap image 102. The dashed circle 1220 encloses one such 
bold line division that contains four higher resolution source pixels. 

FIG. 13 is an expansion of the nine bold line divisions (i.e., nine whole pixels) centered on 
circle 1220 of FIG. 12. The pixel inside circle 1300 represents a single pixel of the intended display 
device. As FIG. 13 makes clear, each bold line division of grid pattern 1210 encloses four pixels of 
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the higher resolution source image. FIG. 13 also illustrates that each pixel of the intended display 
device is made up of three color subpixels,. including a red, a green, and a blue subpixel, labeled 
"R", "G", and "B", respectively. 

FIGS. 14, 15, and 16, respectively, illustrate the positioning of the rectangular window area 
in the source image from which the luminosity of red, green, and blue colored subpixels in the 
intended display device is determined. The area of each such source image window corresponds 
to the area of a whole pixel in the scaled down image centered around the portion of the source 
image corresponding to the subpixel whose luminosity it is being used to calculate. 

Rectangle 1400 of FIG. 14 encloses the area of the source image window used to 
calculate the luminosity of the red subpixel of the lower resolution display device. Similarly, FIGS. 
15 and 16 enclose the source image windows that correspond to the green and blue subpixels of 
the intended display device, respectively. 

As stated above, the luminosity assigned to a subpixel of a given color is determined by the 
following function, or an approximation thereof. The luminosity is set equal to the intensity of the 
supixel's color in each pixel of the source image totally or partially within the subpixel's 
corresponding source image window, times the percent of that window's area covered by each such 
source image pixel. 

In FIGS. 17, 18, and 19, the luminosity of the red, green, and blue subpixels is a function 
of the respective color luminosities of whole or partial source image pixels contained within a 
source image window centered around the portion of the source image corresponding to a given 
subpixel. This is shown in FIG. 17 for the red (R) subpixel, in which window area 1700 is centered 
on the portion of the source image corresponding to that subpixel. Window area 1800 of FIG. 18 
illustrates the same for the green (G) subpixel, and window area 1900 of FIG. 19 illustrates the 
same for the blue (B) subpixel. 

As a result of the shift between the source image windows for each subpixel, the color 
value derived for each subpixel represents the subpixel's corresponding color in a portion of the 
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source image corresponding to the location of each subpixel, itself, rather corresponding to the 
location of its pixel as a whole. As a result, this use of different source image windows for different 
subpixels of a given pixel increases the spatial resolution of the resulting image. 

In the embodiment of the invention shown in FIGS. 17, 18, and 19, the determination of 
which pixels fall within a subpixel's source image window and the percent of that window each 
such pixel covers is made by an approximation based on the percentage of horizontal and vertical 
scan lines covered by such source image pixels. In FIG. 17, the color value of the red subpixel is 
determined as a result of the percentage of a horizontal coverage line 1720 and a vertical coverage 
line 1740 covered by individual source image pixels, times the red color value of each such pixel. 
The same is true, for respective color values, for the scaled image's green (G) subpixel of FIG. 18 
and its horizontal and vertical coverage lines 1820 and 1840 respectively, and the scaled image's 
blue (B) subpixel of FIG. 19 and its horizontal and vertical coverage lines 1920 and 1940, 
respectively. It should be noted that horizontal coverage lines 1720, 1820, and 1920 are intended 
to represent vertical positions just above or below the vertical midpoint of their corresponding 
rectangular area. This is so the coverage line will not exactly equal that location in the source 
image that represents the border between vertical pixels. In the same manner, the vertical 
coverage line 1740 is intended to represent its horizontal position just to the left or the right of the 
horizontal midpoint of the rectangular area 1700. 

The above defined coverage lines represent an embodiment of an aspect of the invention 
that relates to the use of a continuous function(including any reasonably high resolution equivalent 
of a continuous function, such as one having 5 or more bit resolution), to determine the extent to 
which the area of an original image associated with a given color subpixel is covered by a given 
color or shape. In continuous coverage functions, this coverage is determined, not by sampling, 
but rather by a mathematical function that determines boundary locations at which the given 
coverage starts and stops in one or more dimensions, and calculates coverage as a function of 
lengths or areas between one or more such boundaries or between such boundaries and the 
boundary of the source image window associated with a given subpixel. 
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In the embodiment of the invention shown in FIGS. 17,18, and 1 9 and in FIGS. 30, 31 , 
and 32 the calculation of this continuous coverage function is sped up by estimating the area of 
each source image pixel that is in a given subpixel's corresponding source image window area by 
determining the portion of one or more scan lines within the source image window that is covered 
by each of one or more of the source image's pixels within the window. The percent of the total 
length of the window's scanning lines that is covered by a given pixel is multiplied by the value of 
the subpixel's color in that pixel. Such products are summed over all pixels that cover any of the 
window's scan lines to produce the subpixel's color value. This is how a "line coverage" type of 
continuous coverage function can be used to determine the luminosity of a subpixel when creating 
scaled images of color bitmaps. 

FIGS. 20, 21, and 22 illustrate the use of single horizontal and single vertical coverage 
lines within the source image window 2000 associated with the red (R) subpixel in the lower 
resolution display screen. In FIG. 21, the coverage value associated with horizontal scan line 2020 
is the summation of: 

-the red value of the pixel covered by bracket 2120, times the portion (1/3) of horizontal 
scan line 2020 of FIG. 20 covered by bracket 2120, plus 

-the red value of the pixel covered by bracket 2140, times the portion (1/2) of horizontal 
scan line 2020 covered by bracket 2140, plus 

-the red value of the pixel covered by bracket 2160, times the portion (1/6) of horizontal 
scan line 2020 covered by bracket 2160. 

In similar fashion the coverage value associated with vertical scan line 2040 is the 
summation of: 

-the red value of the pixel covered by bracket 2220, times the portion (1/2) of vertical scan 
line 2040 of FIG. 20 covered by bracket 2220, plus 
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-the red value of the pixel covered by bracket 2240, times the portion (1/2) of vertical scan 
line 2040 covered by bracket 2240. 

The total coverage value for the red subpixel is one half of the coverage value calculated 
for the horizontal scan line plus one half of the coverage value calculated for the vertical scan line. 

Similarly, FIGS. 23, 24, and 25 illustrate the use of single horizontal and single vertical 
coverage lines within the source image window 2300 associated with the green (G) subpixel in the 
lower resolution display screen, and FIGS. 26, 27, and 28 illustrate the use of single horizontal and 
single vertical coverage lines within the source image window 2600 associated with the blue (B) 
subpixel in the lower resolution display screen. 

FIG. 29 is highly simplified pseudo-code representation of an algorithm 2900 for deriving 
scaled subpixel-optimized images from a source bitmap image using line coverage of the type 
described above with regard to FIGS. 17 through 28. 

This algorithm performs a loop 2901 for each pixel row of the output image (i.e., the 
scaled, subpixel-optimized image). 

This loop performs an inner loop 2902 for each pixel in its current row. For each such pixel 
the loop 2902 performs a loop 2904 and a function 2914. 

The loop 2904 is performed for each subpixel of the current pixel of loop 2902. It is 
comprised of an interior loop 2906 that is performed for each of the current subpixel's scan lines, 
such as the scan lines shown in FIGS. 17 through 28. 

The loop 2906 includes a function 2908 and a loop 2910. The function 2908 calculate 
each intersection between that scan line and a pixel boundary. Normally, such intersection 
calculations and the other calculations in this algorithm are performed with limited accuracy, such 
as for example 6 to 8 bits of accuracy, to reduce the storage and computational requirements of 
such computation. 
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Then a loop 2910 performs a function 2912 for each portion of a scan line that occurs 
between two scan line ends, a scan line end and a pixel boundary, or two pixel boundaries. 
. Function 2912 adds to a coverage value associated with the current subpixel of the loop 2904 a 
multiple of the percent of that scan line covered by current portion of loop 2910 times the 
component color value of the pixel covering that portion corresponding to the color of the current 
subpixel, all divided by the number of the subpixel's scan lines. 

Once the loop 2904 has calculated the subpixel luminosity value for each subpixel of the 
current pixel, function 2914 sets the current pixel's color value equal to a color having a compound 
RGB value with red, green, and blue component values equal to those calculated subpixel 
luminosity values. 

In different embodiments of the invention different length color values can be used, such 
as 24 bit, 16 bit, or 12 bit color values. Although the system can be used with a limited color 
palette, it works best with true-color colors, which have at least 4 bits of variability for each of the 
red, green, and blue subpixels. 16 bit color, which commonly allocates 5 bits for red and blue and 
6 bits for green (because of the eyes' greater sensitivity to green), provides even better visual 
results. 

Although the embodiment of the invention described above with regard to FIGS. 17 
through 28 makes use of a single horizontal and a single vertical coverage scan line, other 
embodiments of this aspect of the invention may have more scan lines and/or have scan lines in 
orientations other than horizontal and vertical. 

FIG. 30 illustrates the use of two horizontal coverage lines and two vertical coverage lines 
within a source image window 3020 that can be used to estimate the color coverage associated 
with a red (R) subpixel of a subpixel-optimized image. 
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FIG. 31 illustrates the use of two diagonal coverage lines within a source image window 
3120 associated with a green (G) subpixel of a subpixel-optimized image. 

FIG. 32 illustrates the use of two diagonal coverage lines, a horizontal coverage line, and a 
vertical coverage line within a source image window 3220 associated with a blue (B) subpixel of a 
subpixel-optimized image. 

Of course each of the arrangements of coverage lines shown in each of FIGS. 30 through 
31 can be used on either red, green, or blue pixels. 

FIGS. 33 through 38 illustrate that the line coverage method of calculating subpixel 
luminosity values can be applied to a broad range of different scalings between the size of a 
source image and the resulting subpixel-optimized image. This is true because the line coverage 
method measures line coverage at a fairly high resolution, compared, for example, to many 
sampling techniques. This means that it does a relatively good job of measuring the coverage of 
pixels that are only partially in a subpixel's source image window, as will often result when using 
scaling factors that are non-integer ratios. 

In one embodiment of this aspect of the invention a seven bit resolution is used in 
calculating line coverage, which produces satisfactory results. Higher or lower resolutions can be 
used, but it is preferred that the line coverage resolutions be higher than the two to four bit per 
dimension resolution commonly used in techniques that measure coverage by sampling coverage 
within a subpixel's source image window at an array of sixteen (4x4) to two-fifty-six (16x16) points. 

FIG. 33 uses brackets to illustrate the coverage of two horizontal coverage lines by various 
source image pixels within a source image window associated with the blue (B) subpixel for a 
mapping from a source image resolution to a destination pixel-optimized image having half as 
many horizontal pixels and vertical pixels. FIG. 34 does the same for the two vertical coverage 
lines used with such subpixel luminosity calculation scheme. FIGS. 33 and 34 illustrate an integral 
ratio between the number of pixels in the source and the reduced images. 
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FIGS. 35 and 36 illustrate the coverage of horizontal and vertical scan lines, respectively, 
by pixels of the same source image for a scaling factor in which the reduced subpixel-optimized 
image has only about 40% as many horizontal and vertical pixels as the source image. 

FIGS. 37 and 38 illustrate the same for a scaling factor in which the subpixel-optimized 
image has about 66.66% as many horizontal and vertical pixels as the source image. 

It can be seen that the scan line coverage technique shown in FIGS. 33 through 38 
provide an accurate estimate of the percent of each source image window covered by each source 
image at each of different scalings, with relatively little computation. 

FIGS. 39 and 40 illustrate the geometries associated with an "area" type of continuous 
coverage function. In some embodiments of the invention, the percent of a given subpixel's source 
image window covered by each of its associated source image pixels is calculated, not by the line 
coverage approximation described above, but rather by an actual calculation of the area of that 
part of each such source image pixel that lies within the subpixel's source image window. For 
each such source pixel, the component color value of the pixel corresponding to the color of the 
current subpixel is determined. The luminosity value for each subpixel is then calculated by 
summing the multiples of the percentage coverage value and the value of the subpixel's color for 
each source image pixel that appears in its source image window. 

FIG. 39 illustrates the source image window area 3900 associated with a blue (B) subpixel. 
Source pixel 3920 is contained within source image window 3900, as are portions of eight other 
source pixels. The percent of the source image window 3900 covered by a source pixel 3920 is 
calculated by taking the ratio of the area of the hatched portion 4020 of FIG. 40 over the area of the 
whole source image window 4000. Similarly, the percent of the source image window 4000 
covered by the other source pixels contained within it are calculated by taking the ratios of their 
area within the source image window, as indicated by differently hatched areas of the window 
4000, over the total area of that source image window. 
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FIG. 41 provides a highly simplified pseudocode representation of an algorithm 4100 that 
can be used to implement an area coverage function of the type that is discussed with regard to 
FIGS. 39 and 40. 

The algorithm comprises a loop 4102 that is performed for each pixel row in the sub pixel- 
optimized image to be produced. For each such row the loop 4102 performs an inner loop 4104 for 
each pixel in that row. 

This inner loop 4104 is comprised of a loop 4106 and a function 4116. The loop 4106 is 
performed for each subpixel in the current pixel of the loop 4104. This inner loop 4106 is 
comprised of a function 4108 and a loop 41 10. The function 4108 determines which pixels of the 
source image are in the source image window associated with the subpixel, as described above. 
Once this is done the loop 41 10 is performed for each such source image pixel. 

The loop 4110 is comprised of a function 4112 and a function 4114. The function 4112 
calculates the percentage of the subpixel's corresponding source image window area that is 
covered by the current source image pixel of the loop 4110. Then step 41 14 adds to the luminosity 
value being calculated for the current subpixel of the loop 4106, the multiple of the percentage of 
the subpixel's source image window area covered by the current source image pixel, times the 
source image pixel's color component value corresponding to the color of the current subpixel. 

Once the loop 4106 has been performed for each subpixel in the current pixel, function 
4116 sets the current pixel's color value equal to a color having RGB color component values 
corresponding to the red, green, and blue subpixel luminosity values calculated by the loop 4106. 

FIGS. 42 through 53 relate to aspects of the invention relating to bicolor subpixel-optimized 

images. 

A "bicolor" image is one in which individual pixel colors range between two different color 
values. Commonly these two different color values will be black and white, and the pixels of the 
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source and subpixel image will have values limited to black, white, and, in some cases, a grayscale 
value in between. In some embodiments, however, the two different color values can represent 
any uniform foreground and background colors, and colors intermediary between them. Bicolor 
images are often used to represent text, because the display of text is often bicolored, involving a 
foreground color and a background color. But bicolored images can also be used to represent 
other bicolored shapes, bicolored bitmaps, portions of multicolored bitmaps that are bicolored, or 
multicolored bitmaps that are to be represented with bicolored output, such as a grayscale image. 
For example, a multicolor source image can be treated as a corresponding grayscale image, 
merely by treating each of its pixels as having a grayscale value corresponding to the average 
luminosity of each of its three color components. 

The advantage of using such bicolored subpixel-optimized output images is that they often 
can provide a higher spatial resolution than multicolor subpixel-optimized output images. Such 
higher resolution is allowed where the bicolors are black and white, greyscale values, or opacity 
and transparency, because each subpixel can represent both the foreground and background of 
such bicolor pairs equally as well as any other, since each color of each such bicolor pair has equal 
components of red, green, and blue. Except for the need to perform color balancing, as is 
described below, each subpixel's luminosity can be determined as a function of the extend to which 
the portion of the source image corresponding to its own area in the output image is covered by a 
foreground or a background color. This use of a smaller source window, i.e., one corresponding to 
a subpixel's size rather than to a pixel's size, allows a more accurate spatial representation of the 
source image. 

Where the foreground and background colors are not black and white, the resolution 
produced by bicolor subpixel-optimized images will be best if the foreground and background color 
each have red, green, and blue values that are relatively equal in luminosity, but with the average 
luminosity of the foreground and background color as different as possible. In fact, in some 
embodiments of aspects of the invention relating to bicolored subpixel-optimized images one or 
both of the output bicolors are changed from the corresponding input bicolors by being shifted 
toward a corresponding grayscale color to improve the spatial resolution of the output image. 
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The extent to which a subpixel of a bicolor subpixel-optimized output image is to 
display the foreground color is sometimes represented by an alpha, or opacity, value. Such an 
alpha value indicates the extent to which its subpixel's luminosity should correspond to the 
subpixel's color component in the foreground color or in the background color. An alpha value of 
one means the subpixel's color component value should equal the corresponding color component 
in the foreground color. An alpha value of zero means it should equal the corresponding color 
component in the background color. An intermediary alpha values means the subpixel's color 
component value should be a weighted blend of the corresponding color components in both the 
foreground and background colors. Once a subpixel-optimized bitmap is represented in terms of 
alpha values it can be used to represent bicolor images of a given pattern using different 
foreground and background colors. This is commonly used to represent font shapes, since in the 
presentation of fonts the bitmap pattern of a given character-font shape at a given size is often 
displayed with different foreground and background colors. 

In some embodiments of aspects of the invention relating to bicolor subpixel optimizations 
of bitmap images, a scaled subpixel-optimized image of a bitmap image is produced by associating 
a foreground or background bicolor coverage value with each subpixel of the scaled image as a 
function of: (a) the ratio of the foreground or background color for each source image pixel in a 
source image window corresponding to the area of the subpixel; (b) the percent of that window 
covered by each such source image pixel; and (c) a color balancing function that distributes 
subpixel coverage values to reduce color imbalance. In cases in which a bicolor output image is 
being produced for either a grayscale or a multicolor input image, the coverage values calculated 
for individual subpixels can be derived as a function of the whole pixel luminosity of source image 
pixels that cover its source image window. In some embodiments, the extent to which a given 
luminosity value associated with a given subpixel's source image window is distributed to other 
subpixels is a function of extent to which the luminosity value causes a color imbalance. 

FIGS. 42 through 44 relate to a method of determining the luminosity of each subpixel of a 
grayscale bicolored image of a bitmap image. In FIG. 42, rectangle 4200 encloses a window of the 
source image that is associated with the red (R) subpixel of the scaled image. The luminosity to be 
associated with such red (R) subpixel is a function of the whole pixel luminosity of the one or more 

BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 617494-1722 or 617 497-6222 Page 44 



source image pixels that cover the source image window 4200, multiplied, respectively, by the 
percent of the source image window covered by each such source image pixel. Any known method 
for calculating or estimating such coverage percentages can be used. 

In the embodiment illustrated in FIG. 42 source image window 4200 has associated with it 
two horizontal scan lines 4210 and 4220 and two vertical scan lines. FIGS. 43 and 44 illustrate the 
coverage lines for the source image windows 4300 and 4400 for green and blue subpixels, 
respectively. As before, to estimate the extent to which the source image window areas are 
covered by a source pixel, a mathematical function that determines boundary locations at which 
the given coverage starts and stops along each scan line is run. Coverage is calculated as a 
function of the lengths between one or more such boundaries or between such boundaries and the 
boundary of the source image window associated with the given subpixel. This can be done in a 
manner similar to that described above in FIG. 29. 

When calculating bicolor subpixel-optimized images, color imbalances may occur. This is 
because the bicolor methods is attempting to produce an output image that will be perceived by 
users as being made up only of the foreground and background colors, and where the source 
image has intermediary colors, colors in the spectrum between the foreground and background 
colors. But the coverage values of a pixel's individual red, green, and blue subpixels is determined 
by the percent of foreground color in each such subpixel, meaning that the color of individual 
output pixels would often have no relation to the desired bicolor spectrum (usually grayscale), in 
the absence of such color balancing. 

For example, in a grayscale image, if the source image makes a transition from totally 
white to totally black at a location corresponding to the boundary between a red and green subpixel 
in a subpixel-optimized output image, the corresponding pixel in the output image will have a red 
subpixel coverage value that would tend to cause that subpixel to be turned totally on, and green 
and blue subpixels coverage values that would tend to cause those subpixels to be turned totally 
off. This would result in a visible red color for the pixel, even though, in this example, the output 
image is supposed to be a grayscale image. 
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FIGS. 45 through 47 illustrate how a traditional linear color balancing method of a type 
used in the prior art to color balance subpixel coverage values calculated from the rasterization of 
font outlines can be used to color balance coverage values produced from bicolored bitmaps. 

FIG. 45 illustrates a set of grayscale source image pixels under an RGB grid 4600. Grid 
4600 has four pixel areas enclosed in bold line divisions. Each such pixel area is associated with a 
whole pixel in a subpixel addressable screen on which the output image is to be displayed. Each 
pixel area is further divided into three areas associated with the subpixels of the associated pixel 
on the subpixel-addressable screen. Subpixel-associated area 4610 is associated with a red (R) 
subpixel, subpixel-associated area 4612 is associated with a green (G) subpixel, and subpixel- 
associated area 4614 is associated with a blue (B) subpixel. Similarly subpixel-associated areas 
4616 through 4632 are associated with respective display screen subpixels. 

Subpixel-associated areas 4614 through 4630 are covered in whole or in part by source 
image pixels having nonwhite luminosity values corresponding to various degrees of the 
foreground color, which in this case is black. The total black luminosity value (which corresponds to 
one minus the white, or normal, luminance value) of the source image pixels in each of the 
subpixel-associated areas 4614 through 4630 is mapped into corresponding sub-pixel areas in the 
RGB grid 4700 of FIG. 46. The height of the hatched area within each of the subpixel areas 4744 
through 4760 is determined by the total nonwhite luminosity values of the corresponding sub-pixel 
areas 4614 through 4630. 

The bottom half of FIG. 46 illustrates the use of a center-weighted, symmetrical color filter, 
that can be used to distribute the coverage value associated with a given subpixel, such as the 
subpixel 4750 over five subpixels centered around that given subpixel. Three ninths (3/9^) of the 
coverage value of subpixel 4750 is distributed into sub-pixel 4750, itself. Two ninths (2/9 ms ) of the 
coverage value of the subpixel 4750 is distributed into the subpixels 4748 and 4752 that are 
immediately to its left and to its right, respectively. To complete the color distribution of subpixel 
4750, one ninth (1/9*) of its coverage value is distributed into subpixels 4746 and 4754, which are 
two sub pixels to the left and two sub pixels to the right, respectively of the subpixel 4750. 

BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 617494-1722 or 617 497-6222 Page 46 



In general, color balancing distributes color values within a neighborhood of nearby 
subpixels, in which a nearby subpixel is defined to be one within a distance corresponding to two 
whole pixels from the subpixel whose luminosity is being distributed to it, although in many 
embodiments, such as that shown in FIG. 46, the nearby subpixels to which such distributions are 
made are no more than one whole pixel's distance from the subpixel whose luminosity is being 
distributed. 

FIG. 47 illustrates the result of the symmetrical center-weighted color-balancing filter of 
FIG. 46 when it is applied linearly to the coverage value calculated for each of the subpixels 4744 
through 4760 shown in the top half of FIG. 46. 

In FIG. 47 the coverage value associated with each subpixel 4744 through 4760, shown at 
the top of FIG 47, is distributed using a color balance filter that distributes its coverage value in the 
same proportion to its own subpixel and to the two subpixels to the left and right as is shown in 
FIG. 46. The central grid 4802 of FIG. 47 graphically illustrates the size of the contribution that 
such a distribution makes to each of the subpixels 4744 through 4760. The distribution associated 
with each of the given subpixels 4744 through 4760 is centered in a vertical column located directly 
below its respective subpixel. 

RGB subpixel grid pattern 4804 shown at the bottom of FIG. 47 illustrates the luminosity 
value that is calculated for each subpixel 4740 through 4762 by summing all the contributions that 
have been made to it by all of the coverage value distributions illustrated in the center panel 4802. 
To complete the method, the luminosity values of the red, green, and blue subpixels of each pixel in 
grid 4804 are used as the three component color values that specified the color of each such pixel. 

While this linear method does reduce the color imbalance of the scaled image, it does so at 
the expense a substantial reduction in spatial resolution. This can be understood by comparing the 
values in RGB grid pattern 4804 at the bottom of FIG. 47, which represents the subpixel luminosity 
values in the subpixel-optimized output image, to the values in RGB grid pattern 4800 at the top of 
FIG. 47, which represents the luminosity, or foreground color coverage, of the source image pixels 
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in areas corresponding to the subpixels of that output image. As can be seen by FIG. 47, the spatial 
resolution of the output image is smeared relative to the spatial resolution of the source image. 

The present invention includes an innovation that provides similar color balancing of 
subpixel optimized output images, but often with much less smearing of the output image. It does so 
by using a non-linear color balancing filtering method. A method of applying this non-linear filtering 
is illustrated in FIGS. 48 and 49. 

FIG. 48's RGB grid pattern 4900 is a duplication of the RGB grid pattern 4700 of FIG. 46. 
Once again, the total luminosity, or foreground color coverage, values of the source image pixels 
that correspond to an associated subpixel are represented by the height of hatched areas. 

The first phase in this non-linear color balancing method is illustrated in FIG. 48. As before, 
RGB grid pattern 4900 is divided into fourths by the bold line divisions and each fourth, bracketed 
portion 4902, 4904, 4906, and 4908, is associated with a whole pixel of the scaled, or output, image. 
Each of the pixel areas 4902, 4904, 4906, and 4908 is further divided into subpixel source image 
windows corresponding to subpixels in the output image to be produced. For each pixel, a 
determination is made of which of its subpixels' corresponding source image windows has the 
lowest subpixel luminosity, or foreground coverage value. A luminosity (or alpha) value equal to this 
minimum luminosity/coverage value is added to a luminosity/alpha value that is being calculated for 
each subpixel of the output image pixels 4912, 4914, 4916, and 4918 of RGB grid pattern 4910, 
shown in the bottom half of FIG. 48. 

In the top half of FIG. 48 the hatched line 4920 indicates the minimum luminosity/coverage 
value of the pixel area 4902 is zero, since the first two subpixel source image windows of that pixel 
have luminosity values of zero. Thus, the step shown in FIG. 48 sets the luminosity/alpha value for 
the red, green, and blue subpixel areas of pixel 4912 to zero. In like fashion, the minimum 
luminosity value of pixel area 4904 is determined by the value of the red subpixel source image 
window 4922 of the pixel 4904. This minimum luminosity value is mapped into the corresponding 
pixel area 4914 in the bottom half of the figure. Similarly, the minimum luminosity values of pixel 
areas 4906 and 4908 are mapped into pixel areas 4916 and 4918 in the bottom half of FIG. 48. The 
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resulting partially calculated luminosity/alpha values after the completion of this step are 
represented by RGB grid pattern 4910 at the bottom of FIG. 48. 

The second phase of the non-linear color balancing method is illustrated in FIG. 49. In this 
example of the second phase, the portion of the luminosity/coverage value of each subpixel that is 
in excess of its associated pixel's minimum luminosity/coverage value is mapped into the RGB grid 
pattern 4910 by utilizing a color balance distribution filter of the type shown above with regard FIGS. 
46 and 47. 

The pixel grid 5000 at the top of FIG. 49 corresponds to the pixel grid 4900 at the top of 
FIG. 48 (and has the same sub pixels 4740 through 4762) except that it represents the portion of 
each subpixel's luminosity/coverage value (shown in high frequency hatching) that remains after the 
value of the minimum subpixel luminosity/coverage value for the corresponding pixel (shown in low- 
frequency hatching) has been subtracted from it. 

The subpixel grid 5002 in the middle of FIG. 49 corresponds to the similarly shaped pixel 
grid 4802 in the middle of FIG. 47, except that in it only the excess portions of the subpixel 
luminosity/coverage values shown in the top of FIG. 49 with high frequency hatching are distributed 
using color balance filters of the type shown in FIG. 46. As can be seen in this portion of the figure, 
the excess luminosity/coverage value for each subpixel is distributed to its own subpixel, to two 
subpixels to the left, and to two subpixels to the right using the same proportional filter shown in 
FIG. 46. 

The portion of FIG. 49 near its bottom labeled by the numeral 5004 shows the total of such 
excess luminosity/coverage value that is distributed to each subpixel 4740 through 4762 in this 
example of the non-linear method. The total excess luminosity/coverage value calculated for each 
subpixel is added to the minimum luminosity/coverage value that been previously added to that 
subpixel by the step illustrated in FIG. 48, as illustrated at the bottom of FIG. 49, to produce the total 
luminosity/alpha value to be used for each subpixel in the output image. 



BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 617-494-1722 or617 497-6222 Page 49 



To complete the non-linear color balancing process, the luminosity/alpha values summed 
for each subpixel of RGB grid pattern 4910 are used to determine the corresponding red, green, and 
blue color component values of its associated pixel. The red, green, and blue color values of each 
individual pixel in such a display may not be equal, but the total of the red, green, and blue color 
values in any neighborhood of five or so adjacent subpixels of a pixel row should be substantially 
equal, or balanced. 

A comparison of the results achieved by use of linear and non-linear color balance filtering 
methods is illustrated by FIGS. 50, 51, and 52. 

FIG. 50 illustrates the original unfiltered source pixel luminosity/coverage values as first 
mapped into RGB grid patterns 4700 of FIG. 46 and 4900 of FIG. 48. 

FIG. 51 illustrates the result of the non-linear filtering method, as shown at the bottom of 

FIG. 49. 

FIG. 52 illustrates the result of the linear filtering method, as shown in RGB grid pattern 
4804 of FIG. 47. 

As FIG. 51 shows, the output of the non-linear color balancing method more closely 
resembles the original spatial distribution of luminosity/coverage values of FIG. 50 than does the 
result of the linear method, as shown in FIG. 52. The luminosity values produced by the non-linear 
method of FIG. 51 are significantly less blurred, and, thus, provide a higher visible spatial resolution 
than the output produced by the linear method. This is because the non-linear method seeks to 
perform color balance distribution, which has the detrimental effect of blurring spatial resolution, only 
on those portions of subpixel luminosity/coverage values that need such distribution in order to 
prevent color imbalance. This can be seen by comparing the total of subpixel luminosity/coverage 
values distributed to each subpixel using the non-linear method, as indicated by the numeral 5004 
of FIG. 49, with the corresponding total that is distributed to each subpixel using the linear method, 
as indicated by the entire crosshatched area shown for each subpixel at the bottom of FIG. 47. 
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FIG. 53 provides a highly simplified pseudo code description of one implementation of a 
method of producing a subpixel-optimized bicolor output bitmap from a bitmap source image using 
the non-linear color balancing method. 

The algorithm 5300 in this figure is comprised of a loop 5301 that is performed for each 
pixel row in the image. This loop performs two subloops 5302 and 5322 for each pixel row. 

The loop 5302 is performed for each pixel in the current row of the loop 5301 . For each 
such pixel it performs a loop 5304, a function 5314 and a loop 5316. 

The loop 5304 is performed for each subpixel in the current pixel of loop 5302. For each 
such subpixel it performs a function 5306 and a loop 5308. 

The function 5306 determines which pixels of the source image are in a window of the 
source image corresponding to the subpixel's area in the scaled image. This can be performed by 
any known coverage calculation or estimation function, including the ones described above with 
regard to FIG. 17 through 44. 

The loop 5308, comprised of functions 5310 and 5312, is performed for each source image 
pixel that is totally or partially contained within the current subpixel's source image window. 
Function 5310 calculates the percent of the source image window's area covered by the source 
image pixel's area. Function 5312 adds to a luminosity/coverage value calculated for the current 
subpixel, the multiple of the percentage of the window area covered by the source image pixel times 
the source image pixel's average foreground color intensity. 

In cases in which the bicolor image is a grayscale image, the foreground color intensity 
(often referred to as luminosity or luminosity/coverage value above) can correspond to either what is 
commonly referred to as the luminosity, or to the inverse of what is commonly referred to as the 
luminosity, of each whole source image pixel, depending on whether white or black is the 
foreground color, respectively. If the source image is a multi-color image, the average luminosity 
value of each source image pixel's color components can be used to determine a luminosity value 
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that can be used for the source image pixel in calculating its foreground color intensity for the 
purposes of function 5312. 

Loop of 5304 can be used to determine the luminosity/coverage values of the type shown at 
the top of FIGS. 46. 

Once the loop 5304 is been performed for each subpixel in the current pixel, function 5314 
finds the minimum subpixel luminosity/coverage value that has been calculated for the current pixel, 
as is illustrated in the top half of FIG. 48. 

Once this has been done, the loop 5316, comprised of functions 5318 and 5320, is 
performed for each subpixel in the current pixel. 

Function 5318 sets a luminosity/alpha value being calculated for the subpixel to the 
minimum subpixel luminosity/coverage value determined for the pixel by the function 5314, as is 
indicated in the bottom half of FIG. 48. 

Function 5320 distributes the portion of the subpixel luminosity coverage value that exceeds 
the pixel's minimum subpixel luminosity/coverage value to the luminosity/alpha values being 
calculated for the subpixel and adjacent subpixel's in the current pixel row using a color balance 
distribution filter, as is indicated in FIG. 49. 

In one embodiment of the invention, if the total of such distributions made to a given 
subpixel exceeds the maximum allowed luminosity/output value, the subpixel's luminosity/alpha 
value is limited to that maximum value. Although this clipping causes some color imbalance, the 
inventors have found the resulting imbalances to be hardly noticeable. 

Once luminosity/alpha values have been calculated for each subpixel in the row and the 
loop 5302 has been completed, the loop 5322 causes a function 5324 to be performed for each 
pixel in the row. This function sets the pixel color value equal to a color having a compound RGB 

BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 617-494-1722 or 617 497-6222 Page 52 



value with red, green, and blue component values corresponding to the luminosity/alpha values 
calculated for the pixel's red, green, and blue sub pixels, respectively. 

FIG. 54 illustrates an aspect of the present invention in which the user of a display device 
can dynamically make tradeoffs between the extent to which a subpixel-optimized image produced 
from a source images is produced by a multi-color subpixel optimization process or a bicolor 
subpixel optimization process. An output image produced by the multi-color subpixel optimization 
can represent scaled-down color images with reasonable color accuracy, whereas the bicolor 
subpixel optimization in this example can produce only grayscale output images. But in some 
cases, such grayscale output images will have a more accurate spatial resolution and, particularly 
where the source image has black and white portions with sharp edges, less perceptible color 
imbalance than an output image produced by the multi-color subpixel optimization. 

A color bitmap image 5400 may be scaled and subpixel optimized by utilization of functions 
5410 and 5430, which use a bicolor subpixel-optimization method, such as one of those described 
above with regard to FIGS. 42 through 53, to produce a scaled and subpixel-optimized grayscale 
bitmap 5440. Color bitmap image 5400 may also be scaled and subpixel optimized by a process 
5420 that uses a multicolor subpixel-optimization method, such as one of those described above 
with regard to FIGS. 17 through 41 , to produce a scaled and subpixel-optimized color bitmap 5450. 

According to an embodiment of the present invention shown in FIG. 54, the user of the 
display device can manipulate a control apparatus of the display device, such as a pointing device, 
keyboard, or other input device, using function 5460 in order to achieve a blend of the color bitmap 
5450 and the grayscale bitmap 5440. The can be done, for example, by manipulation of a slide bar. 
Function 5480 receives the grayscale bitmap 5440 and the color bitmap 5450 and the user selected 
color/grayscale tradeoff information and blends the color values of the corresponding pixels from the 
grayscale and color bitmaps 5440 and 5450, weighing color values from each as a function of the 
user selected color/grayscale tradeoff 5460. 
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In some embodiments of the type shown in FIG. 54, if the user selects a tradeoff value at 
either extreme of the color/grayscale spectrum, the process can reduce computation by only 
calculating the bitmap 5440 or 5450 that corresponds to that selected extreme. 

A benefit of this aspect of the present invention is that the user of a display device can favor 
color balance and/or positional accuracy when that is most important or color accuracy when that is 
most important or simply vary the tradeoff selection to find a more easily readable display. 

Not all aspects of the present invention require subpixel-optimized text, and many of those 
that do can use prior art methods of creating subpixel-optimized bitmap's of font shapes. However, 
some aspects of the invention relate to improved methods of making subpixel-optimized font 
bitmaps. 

FIGS. 55 through 97 relate to aspects of the invention relating to the forming and using of 
subpixel-optimized font bitmaps. 

FIG. 55 illustrates a font outline 5500, in this case an outline of a capital letter "B" in a 
Times Roman font. The outline is shown superimposed over a subpixel grid 5502, which is 
composed of a plurality of individual whole pixels 5504, each of which includes a red, blue, and 
green subpixel, 5506, 5508, and 5510, respectively. 

The font outline shown in FIG. 55 is one that could be used for display at relatively normal 
text sizes, indicating that the invention's method of subpixel optimizing character-font shapes is 
applicable across a broad range of applications and is not limited to small screen displays of the 
type shown at the bottom of FIG. 1 1 . However, when this aspect of the invention is applied to 
small screen displays and/or the display of font at very small pixel sizes, it is preferred that the 
fonts used be optimized for display at small sizes, such as in some embodiments for display at ten 
pixels per em or less, and in other embodiments for display at eight pixels per em or less. 

FIG. 56 illustrates a font that has been optimized for such small display on subpixel 
addressable screens. FIG. 57 shows the same bitmap at twice the size. Unfortunately the 
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printouts of the bitmap shown in FIGS. 56 and 57 display the average luminosities of whole pixels 
and fail to capture the higher resolution made possible when such a bitmap is shown on a subpixel 
addressable display. 

The font shown in FIGS. 56 and 57 have been produced by a hinting process that shifts 
selected boundaries of individual font outlines to pixel boundaries, subpixel boundaries, and 
horizontal and vertical dimension's intermediary between subpixel boundaries. Such high resolution 
hinting is used in order to achieve optimum readability on subpixel displays. It is done by having a 
font designer view subpixel-optimized bitmaps of individual characters with various hinting values 
until he or she feels relatively satisfied that the character is as clear as possible when displayed at 
such a small font size. As those knowledgeable of font hinting will understand, a font can have 
hints that dictate the alignment of individual portions of a font outline across all size renderings of 
that font, and special hints that are applied for the character-font shape at certain pixel sizes. The 
font shown in FIGS. 56 and 57 have been hinted to optimize their display at eight pixels per em, 
and some of them have specific hints that are to be applied only at such small sizes. 

In fact, most of the fonts in the 320 by 240 pixel resolution screen shots shown in the 
figures of this application are of 8 pixel per em fonts that have been specifically hinted for display at 
that size. These fonts allow a relatively large amount of web text to fit on a small screen, while 
allowing a relatively high level of readability. These fonts allow the large majority of lower case 
characters of the Latin alphabet to be represented in four pixel columns or less, including the space 
that separates adjacent characters, if any. These fonts allow a majority of capital characters to be 
represented in 5 pixel columns or less. 

The readability of such small fonts is greatly increased by the use of the technique of either 
subpixel optimization or anti-aliasing, because these techniques allows information about the 
extent to which a character shape covers a given pixel to be represented at more than just a binary 
representation at the whole pixel level. In fact, subpixel optimization is a type of anti-aliasing 
because it, like traditional anti-aliasing causes pixels that are partially covered by a font shape to 
have color values that vary as a function of the extent of such coverage. 
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FIG. 58 and 59 illustrate that subpixel-optimized bitmaps produced by the present 
invention can be represented as font outlines and/or font bitmaps. The font outline descriptions 
5802 contain a mathematical geometric description of the shapes of one or more characters in a 
given font, preferably with hinting information designed to optimally place the boundaries of 
character outlines at one or more different font sizes. These font outlines can be ones, such as 
those just discussed, that have been designed to be rendered optimally on a subpixel addressable 
display and/or have hinting which has been optimized for display on a subpixel addressable 
display. 

A font Tenderer 5806 can be used to create a subpixel-optimized bitmap 5804 from such 
outlines, as is described below. 

In some embodiments of the invention, illustrated in FIG. 58, a computer 5808 and/or an 
application which is running on that computer display text using font bitmaps or font outlines 
accessed over a computer network 5814 from a font server 5812. In other embodiments, 
illustrated in FIG. 59, a computer 5900 and/or an application 5902 running on it have font bitmaps 
5804 necessary to render text stored within them. Such computers and/or applications can store 
only font bitmaps, or they can store scalable font outlines 5802 and render font bitmaps 5804 as 
needed at different sizes. 

The advantage of storing only font bitmaps is that it prevents the need to store font outlines 
and a font renderer on the computer 5900. It also prevents the need for the computation involved 
in font rendering. Furthermore, many font vendors are more willing to allow font bitmaps to be 
relatively freely available over the Internet then they are font outlines. 

The advantage of storing font outlines is that if one is interested in rendering fonts at a 
large variety of sizes, it is actually more efficient to store the code necessary for the font renderer 
and to store scalable font outline descriptions than it is to store font bitmaps for all the differently 
sized characters. 
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The advantage of receiving fonts from a font server as shown in FIG. 58 is that it allows a 
client computer 5808, such as that represented in FIG. 58, to represent text in any one of a large 
number of different fonts and sizes by downloading such fonts as needed, without the need to store 
a large library of fonts. Preferably the client computer 5808 will cache a reasonable number of 
character-font bitmaps so that there is no need to communicate over the network 5814 every time it 
seeks to display a string. 

FIG. 60 is a highly simplified pseudocode description of an algorithm 6000 used by some 
embodiments of aspects of the invention relating to producing subpixel-optimized font bitmaps. 
This algorithm uses non-linear color balancing of the type described above with regard to FIGS. 48 
and 49. Such a subpixel optimization algorithm is particularly optimal for use in the display of text 
characters, because the alignment of text outline boundaries with whole pixel boundaries is quite 
common in rasterized font shapes because of the use of hinting. 

The algorithm 6000 of FIG. 60 includes a loop 6002 that is performed for each pixel row in 
the rasterization of an individual character-font shape at a given pixel resolution. This loop 6002 is 
comprised of three subloop's 6004, 6008, and 6020, that are sequentially performed for each pixel 
row. 

The loop 6004 is performed for each subpixel in the pixel row for which the current iteration 
of the loop 6002 is being performed. For each such subpixel, the loop 6004 performs a function 
6006, that determines the coverage value for each such subpixel as a function of the percent of the 
subpixel's area covered by the character-font shape of which an image is being made. 

FIGS. 61 through 90 are used to discuss methods that can be used to determine the 
coverage value of each subpixel in step 6006 of FIG. 60. 

As is indicated in FIGS. 61 , 62, and 63 the area in the image of the character-font shape 
for which such a coverage value is calculated for a given pixel 5504 corresponds to the area of that 
image that will be displayed by each red, green, and blue subpixel 5506, 5508, and 5510, 
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respectively. This is different than in the case of subpixel-optimized multicolor images, in which the 
source image window corresponding to each subpixel is normally larger, as is indicated in FIGS. 14 
through 16 above. The source image window used by the method of FIG. 60 has the same sizes 
as the area of the source image window used for bicolor bitmaps described above regard to FIGS. 
42 through 44. 

Such a higher resolution source image window can be used because the character-font 
shapes described by most font outline descriptions are bicolor images, with the area covered by 
the font outline considered as being associated with a foreground color (in most cases, 
represented by an alpha value of one) and all other portions of the image being associated with a 
background color (in most cases, represented by an alpha value of zero). 

The calculation of the coverage values in function 6006 of FIG. 60 can be performed using 
any prior art technique capable of rasterizing a character font outline relative to an array of pixels 
having the same spatial resolution as the subpixel's of the grid 5502 shown in FIG. 55. 

FIGS. 64 through 67 illustrate some of the traditional methods that have been used to 
calculate the percentage of a unit in a rasterization grid that is covered by a font shape 6402. In 
the prior art, the unit of rasterization 6400 has typically corresponded to a whole pixel in the output 
image. In the method of FIG. 60 it corresponds to a subpixel in the output image. 

FIG. 64 illustrates one method of determining the coverage of a rasterization unit 6400 that 
uses mathematical techniques to exactly calculate the area of the unit that is covered by the font 
shape 6402. This is relatively computationally expensive, and thus is hardly ever used. 

A substantially more computationally efficient method is shown in FIG. 65, which calculates 
the percentage of the rasterization unit 6400 that is covered by the font shape 6402 by using 
piecewise linear approximations 6504 of the boundary of the font shape. 

FIG. 66 illustrates an even more computationally efficient manner, although it produces a 
substantially less accurate results. This method determines the percent of coverage of the 
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rasterization unit 6400 by determining what percent of a set of sample points 6600 fall inside the 
shape 6402. 

FIG. 67 illustrates a method of determining coverage values that provides more accurate 
results for the same, relatively low degree of computation as the method of FIG. 66. It determines 
the coverage of the rasterization unit as a function of the average percentage of a number of scan 
lines 6700 and 6702 that are covered by the font shape 6402. 

FIGS. 68 through 87 illustrate an extremely computationally efficient method of calculating 
the coverage of a rasterization unit, which yields results that are typically better than a sampling 
method such as that shown in FIG. 66 for the same amount of computation. 

An embodiment of this method is described in much more detail in a U.S. patent 
application filed in the name of one of the inventors of the present application, Sampo J. Kaasila. 
This U.S. Patent application has the serial number 09/363,513. It was filed on July 29, 1999, and 
is entitled "Systems For Rapidly Performing Scan Conversion With Anti-Aliasing Upon Outline 
Fonts And Other Graphic Elements". This application issued as U.S. Patent 6,437,793 on August 
20, 2002. This application also has had its disclosure published in PCT application 
PCT/US00/21559. This application and the patent that has issued from it are hereby incorporated 
herein by reference in their entirety. 

In the method of FIGS. 68 through 87, the coverage value for a rasterization unit is 
determined by that percentage of one of its two scan lines, a horizontal scan line 6804 or a vertical 
scan line 6802, that is covered by a font outline's shape 6402. The scan line whose coverage 
value is used as the coverage value for the rasterization unit is that which has the more 
intermediate coverage value. For example, in an embodiment where the coverage for the 
horizontal and vertical scan lines is calculated in a range of values from 0 to 126, the scan line 
chosen is that whose value is closest to 63, which represents a 50 percent coverage. 

In FIG. 68 through 71 it is the percentage of coverage of the vertical scan line 6802 that is 
used to represent the percentage of coverage of the rasterization unit 6400. In FIGS. 72 through 
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75 it is the horizontal scan line 6804 that has the most intermediate values, and, thus, which has its 
percentage of coverage used to represent the percentage of actual coverage of the entire 
rasterization unit. 

In all the rest of the FIGS. 76 through 87 it can be seen that the coverage value of the scan 
line with the more intermediaries coverage value normally is very close to the actual coverage 
value for the entire rasterization unit, and that it normally never varies from the actual coverage 
value of the entire rasterization unit by more than 25 percent. 

FIGS. 88 through 90 represents other combinations of scan lines that can be used 
according to a method that weighs the contribution of the coverage values of individual scan lines 
to the estimated coverage value of their associated rasterization unit as a function of which of 
those line coverage values have more intermediate coverage values. In such methods the 
coverage value calculated for an entire rasterization unit can be set equal to the sum of the 
coverage value of each scan line times that coverage line's mediality, all divided by the sum of 
each scans line's mediality. In this calculation, a scan line's mediality equals the scan line's 
middlemost possible percentage coverage value minus the absolute value of the difference 
between that middlemost percentage coverage value and the scan line's actual percentage 
coverage value. 

FIG. 91 illustrates a hypothetical font outline 9102 mapped over the red, green, and blue 
subpixels 5506, 5508, and 5510, respectively, of a row 9100 of pixels 5504. 

FIG. 92 illustrates the corresponding coverage values 9202 that have been calculated for 
each of the subpixels in the row 9100. 

FIG. 93 illustrates how the coverage values determined for an individual subpixel can be 
distributed using a linear color balance method. This linear color balancing is identical to that 
described above with regard FIG. 46, except that it is being applied to the subpixel-optimized 
representation of a font outline rather than a subpixel-optimized representation of a bitmap source 
image. 
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Returning briefly now to FIG. 60, once step 6006 of that figure has calculated or estimated 
the coverage value for each subpixel of a row, as indicated in FIG. 92, a loop 6008 is performed for 
each pixel in the row. This loop color balances the coverage values calculated for the subpixels of 
a row. It does not use a linear color balancing routine of the type illustrated in FIG. 93 and 
described above with regard to FIGS. 46 and 47. Instead it achieves higher perceivable spatial 
resolution by using a non-linear color balancing technique similar to that described above with 
regard to FIGS. 48 through 53. 

The loop 6008 performs two functions, 6010 and 6012, and a loop 6014 for each such 

pixel. 

The function 6010 finds that subpixel of the current pixel has had the minimum coverage 
value calculated for its subpixel. Then step 6012 adds this minimum coverage value to the 
temporary alpha, or opacity, value being calculated for each subpixel of the current pixel. This is 
similar to the function illustrated in FIG. 48. 

Then a loop 6014 performs function 6016 and 6018 for each subpixel of the current pixel. 
The function 6016 determines, for the current subpixel of the loop 6014, the excess of the 
coverage value that has been calculated for it over the minimum coverage value that has been 
found for the pixel of which the subpixel is part. Then function 6018 distributes this excess value 
across the subpixel alpha values being calculated for the current subpixel and the two subpixels to 
its left, and the two subpixels to its right in the current pixel row. This function corresponds to that 
described above with regard to FIG. 49. 

FIGS. 94 and 95 illustrate two different color balance distribution filters that are used in one 
embodiment of the present invention. In this embodiment a symmetrical center weighted color 
balance filter shown in FIG. 94 is used to distribute the coverage values associated with the red 
and green subpixels. On the other hand an asymmetrical color balance filter shown in figure 95 is 
used to distribute coverage values associated with blue subpixels. Thus, this embodiment of the 
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invention differs from the process described above with regard to FIG. 49 in that it used differently 
shaped distribution filters for some colors than for others. 

One of the inventors of the present application has found that because the eye perceives 
green much more strongly that it does blue, that color balancing coverage values associated with 
differently colored subpixels should use such different distribution filters. In different embodiments 
of the invention relating to non-linear color balancing (including the non-linear color balancing of bi- 
color images) a different color balancing filter could be used for each different subpixel color, the 
same color balance filter could be used for all such colors, and either symmetrical or asymmetrical 
color balancing filters can be used. 

The particular color-balancing filters shown in FIGS. 94 and 95 are designed for use with 
coverage values that are calculated on a scale from 0 to 126. A given calculated coverage value 
having a value from 0 to 126 is associated with that one of the set of five distribution values on the 
right hand side of the tables of FIGS. 94 and 95 whose associated coverage value on the left side 
of that table is closest to its own coverage value. For example, if the coverage value calculated for 
a given red or blue subpixel was 126, an addition of 1 would be made to the alpha value being 
calculated for subpixels two to the left and two to the right of the given subpixel, an addition of 3 
would be made to the alpha values being calculated for the subpixels one to the left and one to the 
right of the given subpixel, and a value of 4 would be added to the alpha value being calculated for 
the given subpixel, itself. In this particular embodiment the alpha values are calculated on a scale 
from 0 to 12. 

The relative size of the color balance distribution shown in the last row of FIGS. 94 and 95 
reflect more accurately the desired distribution ratios. This is because the larger value distributed 
in each of these last rows allows greater numerical resolution than is found in the rows above each 
of them. 

It should be appreciated that in some other embodiments that use higher numerical 
accuracy to describe the coverage or luminance values being balanced, the balancing distributions 
used across a broad range of luminosities would have ratios between the contributions to different 
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subpixels more like those reflected in these last rows of FIG. 94 and FIG. 95. This is particularly 
true when filters of the general type shown in FIG. 94 and/or FIG. 95 are used in the color 
balancing of bicolor subpixel optimizations of images, such as is described above with regard to 
FIGS. 48 through 52. This is because, in such bicolor subpixel optimizations of bitmap images, it is 
commonly not desirable to compute the luminance of output images at a color resolution lower than 
that used in the bitmap being subixel optimized. 

Once loop 6008 of FIG. 60 has caused step 6018 to be performed for each subpixel of 
each pixel in a row, each such pixel will have a separate alpha value calculated for each of its three 
subpixels, with each such alpha value having one of thirteen opacity levels. This means it is 
possible for each pixel to have one of 2,197 (i.e., 13 3 ) different possible combined alpha values. In 
other embodiments of the invention alpha values with higher or lower resolution can be used. 

In many embodiments of the invention, particularly those designed to run on computers 
with limited computational capacity or in systems in which it is desirable to reduce the bandwidth or 
storage capacity required to store or download font bitmaps, it is desirable to map from the 
relatively large color space of the 2,197 combination of different subpixel alpha values possible 
after such color balancing into a smaller color space. 

The embodiment of the invention in FIG. 60 performs such a mapping in loop 6020. Once 
the loop 6008 has been performed for each pixel in the current row, the loop 6020 performs an 
additional function 6022 for each such pixel. The function 6022 takes the three alpha values that 
have been calculated for each of a pixel's subpixels and uses them as an input value of a lookup 
table that maps from each of the 2,197 possible color value defined by the possible combination of 
a pixel's three alpha values into 1 of 122 values. In this embodiment the color space has been 
reduced down to such a small number of colors so that a machine that has a 256 value color space 
will be able to display each of the 122 values selected for use in the display of subpixel optimize 
fonts while still having over half of such a limited color space for other uses. The uses of such a 
small color palette to represent font bitmaps reduces the number of bits required to store such font 
bitmaps and makes them more efficient to download. In other embodiments of this aspect of the 
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invention the source and the destination color spaces used in such a mapping could have different 
sizes. 

FIG. 96 illustrates the method 9600 that has been used to create such a color mapping in 
one embodiment of the invention. It is to be understood that in other embodiments, other types of 
mapping could be used. In some embodiments no such mapping into a smaller color space need 
be used at all. 

The method of FIG. 96 starts with a step 9602 that runs multiple characters from the one 
or more font whose bitmaps are going to be represented using the limited color space being 
calculated through the non-linear color-balanced subpixel-optimization algorithm described above 
with regard to FIGS. 60 through 95. When this is done a histogram is kept of the number of times 
each of the possible 2,196 different composite pixel alpha values is calculated for any of the pixels. 
This histogram is useful because most of the three-colored alpha values calculated for pixels in 
subpixel-optimized font bitmaps tend to be concentrated into a various small regions of the total 
possible color space of 2,196 such three-color alpha values. This concentration is probably even 
more pronounced with non-linear color balancing, because it substantially reduces the amount 
luminosity distributions due to color balancing. 

Next a function 9604 creates a limited color palette, in this case having 122 colors, by 
performing the functions 9606 and 9608. The function 9606 selects, as part of the palette being 
formed, the thirteen grayscale values that are possible for whole pixel alpha values, given that 
each subpixel can have one of thirteen alpha levels. Then the function 9608 selects the 109 other 
most frequently occurring colors in the histogram previously calculated by step 9602. 

Once the limited color palette has been selected, a loop 9610 is performed for each of the 
2,196 possible whole pixel alpha values. For each such possible alpha value a conditional 9612 
tests to see if that input color exactly matches one of the 1 22 colors. If so, the function 9614 
associates the input color with its identical output color in the lookup table being constructed. If the 
condition 9612 is not met, a loop 9618 and a function 9628 will be performed for the current input 
color of loop 9610. 
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The loop 9618 is performed for each of the 122 output colors in the palette. It has a 
conditional 9620, that tests to see if the difference (rko) between the red alpha value of the input 
color to be mapped and the current output color of the loop 9618 is of the same sign as the 
difference (gi-go) between the green alpha value of the current input color and the green output 
alpha value for the current output color. The conditional 9620 also tests to see if the difference (ro- 
go) between the red alpha value and the green alpha value of the current output color is less than 
the difference (ri-gi) between the red alpha value and the green alpha value of the input color (plus 
a possible value X to allow some leeway). If these two conditions, which are designed to prevent 
relatively noticeable differences between an input color and the output color to which it is to be 
mapped, are met, functions 9622 through 9626 will be performed. 

Function 9622 calculates the distance from the input color to the output color. Function 
9624 tests to see if that distance is the closest distance so far to the input color in the current loop 
9618. If the test of function 9624 is met, step 9626 saves the current output color of the loop 9618 
as the closest allowed palette color. After the loop 9618 has been performed for each of the 122 
output colors of the limited palette, step 9628 associates the current input color of the loop 9610 
with the closest allowed palette color calculated in the loop 9618. 

Once the loop 9610 has been performed for each of the possible input colors, each of 
those input colors will have been mapped to one of the 122 output colors. 

In the particular color mapping scheme shown in FIG. 96 non-grey scale pixel color values 
produced by color balancing get mapped in to grayscale color values if they do not get mapped into 
one of the one hundred and nine most frequently occurring non-grayscale color values selected by 
step 9608. This generally yields results at least as good as traditional anti-aliasing, which 
represents all bitmaps with a grayscale alpha value. 

FIG. 97 illustrates an algorithm 9700 used to display font bitmaps of a type generated by 
the methods of FIG. 60 and 96 on a subpixel addressable screen. 
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The loop 9702, comprised of the function 9704 and loops 9706 and 9714, is performed for 
each string to be displayed. 

Function 9704 samples a set of points in the rectangle of the bitmap at which the string is 
to be drawn to determine the average background color value for the string. In other embodiments 
the background color is separately determined for each character or for each pixel of each 
character, but in the embodiment shown, the background color is determined only once for each 
string to save computation. 

Once the background color for the string has been determined, loop 9706 performs a 
subloop 9708 and a function,9712 for each of the 122 whole pixel alpha values, described above 
with regard to FIG. 96. 

The loop 9708 performs a function 9710 for each of the three subpixel colors. The function 
9710 calculates the luminosity value for the current subpixel color as a function of the components 
of the current whole pixel alpha value corresponding to that current subpixel color. It sets the 
luminosity value it is calculating equal to this subpixel alpha value multiplied by the luminosity of 
the current subpixel's corresponding color in the foreground color of the string to be drawn, plus a 
quantity of one minus the current subpixel's alpha value multiplied by the luminosity of the current 
subpixel's corresponding color in the background color determined by function 9704. 

Once this loop has been performed for each of the three subpixel colors, function 9712 
maps the current whole pixel alpha value of the loop 9706 into the whole pixel color value 
comprised of the three subpixel luminosities that have just been calculated in the loop 9708. 

Then the loop 9714 performs the function 9716 and the loop 9718 for each of the 
characters of the current string to be displayed on a subpixel addressable display. 

Function 9716 accesses the font bitmaps for the current character. Then the loop 9718 
performs functions 9720 and 9722 for each pixel of that bitmap. Function 9720 finds the color 
value that has been mapped by the loop 9706 into the current whole pixel alpha value indicated for 
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the current pixel in the character's font bitmap. Once this color value has been found, function 
9722 sets the corresponding pixel in the subpixel addressable display to the that whole pixel color 
value. 

Once the loop 9718 has been performed for each pixel of each character of the string, the 
string will have been completely displayed in a subpixel optimize manner. 

FIGS. 98 through 101 are used to illustrate how well the techniques for image and font 
scaling and subpixel optimization work. FIGS. 98 and 100 illustrate views of two different web 
pages laid out and displayed at 640 by 480 pixels using a common browser program. FIGS. 99 
and 101 illustrate the same web pages after their images and text have been scaled by the method 
described above so as to fit on a 320 by 240 display. Unfortunately, the 320 by 240 pixel images 
are printed with grayscale values determined by the average luminosity of its whole pixels, and 
thus the actual clarity added by subpixel resolution is not shown in these images. 

FIGS. 102 through 113 illustrate in more detail the interaction between a proxy server and 
a thin client computer in one embodiment of the present invention. 

FIG. 102 is a highly schematic block diagram of a system including a proxy server 210 and 
a thin client 200 of the type described above with regard to FIG. 2. 

The proxy server 210 includes a browser 10200 that includes programming 10202 to 
perform the standard functions of a full Web browser. This programming has been modified 
because the browser operates as a proxy for the thin client. When the proxy browser receives over 
the network an HTML description 10204 of a requested web page, it creates a two dimensional 
layout 10206 of that web page. 

FIG. 103 illustrates a portion of the HTML description of the web page whose display is 
shown in FIGS. 98 and 99. The numerals 10300 shown in FIG. 103 illustrates portions of text in 
the HTML that are shown in the left-hand column of the web page shown in FIGS. 98 and 99. The 
numeral 10302 points to an image tag that identifies the bitmap used to represent the word 
"Sections" shown in the same column. 
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When the proxy browser code receives the download of the web page, it attempts to 
create a layout 10206 of that web page at a virtual layout resolution. The virtual layout resolution 
often corresponds in width to a virtual screen size, which is the size of the window into which the 
proxy browser thinks it is displaying all or a portion of the web page. We call this window into 
which the browser thinks it is displaying the web page the virtual screen 10208. 

FIG. 104 illustrates the layout 10206 of the web page shown in FIGS. 98 and 99, and it 
shows in heavy black rectangle 10208 the mapping of the virtual screen into that layout. 10220 
shows the actual screen image that is displayed on the thin client given the location of the virtual 
screen shown in FIG. 104 to the layout 10206. 

Many web pages today include elements larger than the 640 by 480 virtual screen 
resolution used in the example system being described. The layout will have the minimum pixel 
width required to layout the objects of the web page, or the width of the virtual screen, which ever 
is larger. For example, it is common today for many web pages to be laid out with a minimum 
possible width resolution of 800 pixels. In this case the virtual screen will have a smaller pixel 
width than the layout. This is the case in the example shown in FIG. 104. The height of the virtual 
layout is that needed to display all of the web page, given the virtual layout's width. 

The view window 10210 shown in FIG. 102 represents that portion of the virtual screen 
that is to be actually displayed upon the screen of the thin client. In views shown in FIGS. 99, 101, 
and 104 the view window equals the virtual screen. But if the user zooms in on a portion of the 
virtual screen, the zoom's scale factor control 10216, shown in FIG. 102, will change and the view 
window will be mapped into a subset of the virtual screen. 

Scroll control 10218 shown in FIG. 102 causes the view window to move relative to the 
layout. If the view window is moved so that it includes a portion of the layout that is not in the 
current location of the virtual screen, a command will be sent to the browser software to scroll the 
virtual screen. The event queue 10220 stores events, that is, user input, that has been received on 
the thin client and that have been uploaded to the proxy server for corresponding action by the 
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browser. Events that occur on the screen of the thin client are mapped through the view window to 
the corresponding locations on the virtual screen and then placed in the event queue of the proxy 
browser, so the proxy browser will respond to such input as if it had been received at the 
appropriate location on the virtual screen it thinks it is drawing directly onto a video output device. 

The browser programming 10202 of FIG. 102 has been modified so that each time it thinks 
it is drawing an object on the virtual screen it creates a corresponding scaled-down object at a 
correspondingly scaled location in a download display list 10212. 

This display list is downloaded over the network 10222 to the client computer, which stores 
it, as is indicated by the numeral 10212A. The scaled down images 10214 referred to by this 
display list are also downloaded. Programming 10218 located on the thin client displays the 
strings, images, and other elements contained in the display list on the thin client screen 10221 . If 
the user clicks on the thin client screen, the operating system 10222 of the thin client places such a 
click and its location on the thin client's screen in an event queue 10224. Each such event that 
does not relate to programming handled locally on the thin client is uploaded to the event queue 
10220 of the proxy server, as described above. 

FIGS. 105A through 1 10 are highly simplified pseudocode descriptions of programming 
and data structures on the browser and thin client computers designed to control their interaction 
for the purpose of allowing the thin client to browse web pages through the proxy. 

FIGS. 105A and 105B are highly simplified pseudocode representations of portions the 
browser's code 10202 shown in FIG. 102 used to help it function as a proxy for the thin client. 

In the particular embodiment illustrated in these figures, a large Web browser designed for 
normal use has been patched to make it perform as a proxy. It is to be understood that in other 
embodiments of this aspect of the invention the functionality necessary to make the browser 
operate as a proxy could be more intimately and elegantly integrated into the browser's code. In 
yet other embodiments, code in the operating system, or in functions that intercept operating 
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system calls can be used to make a standard Web browsing program operate as a proxy for a thin 
client. 

In the embodiment shown in FIG. 105A, if the proxy browser receives a request from the 
thin client for a web page, steps 10502 and 10504 relay that request to the server computer 
indicated in the URL of the request. 

If the browser receives an indication from its own code that the browser has completed a 
draw or redraw of the virtual screen 10208 described above with regard FIG. 102, functions 10506 
and 10510 will call the screen capture and download routine shown in FIGS. 106A through 106C. 

FIGS. 106A through 106C are highly simplified pseudocode descriptions of the screen 
capture and download routine 10600. 

When this routine is called by function 10510 of FIG. 105A, just described, its step 10602 
asks the browser for a screen redraw, which causes the browser to call routines to draw each of 
the elements in the web page's layout that all or partially fit within the virtual screen. The routine of 
FIGS. 106A through 106B records information contained in each of these draw calls and uses it to 
create the download display list 10212 shown in FIG. 102. 

If the browser calls a measure string routine 10606 of FIG. 106A, this routine causes 
functions 10608 through 10618 to be performed. Such calls are made by the browser to determine 
the size of text it is seeking to layout into the virtual screen. Although not shown in the figures, 
these same functions 10608 through 10618 are performed anytime the browser makes a call to 
measure string size, even if it is not during the operation of the screen capture and download 
routine shown in FIGS. 106A through 106B. This causes all virtual layouts of a web page to use 
font metrics supplied by function 1068 through 10618. 

Function 10608 maps the font specified in the measure string call into a font having a 
different font family and a different font size. This font substitution is controlled by three 
considerations indicated by numerals 10610 through 10616. 
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Consideration 10610 seeks to select a size for the substitute font as a function of the 
requested font size in the call to the measure string routine and the display scale factor 

The display scale factor is a ratio of the resolution along a given dimension of the portion 
of the virtual screen 1028 corresponding to the view window and the resolution, along the same 
dimension, at which the view window will be displayed on the thin client. In some cases the display 
scale factor will have different components to represent different scaling ratios to be used along the 
horizontal and vertical directions, but in many cases the display scale factor will be comprised of a 
single scaling ratio to be used for both horizontal and vertical resolution. 

In the particular embodiment being described, when the layout of a web page is performed 
it is performed for an un-zoomed display scale factor. An un-zoomed display scale factor is one in 
which the view window 10210, of FIG. 102, is equal in size to the virtual screen 10208. As a result, 
an un-zoomed display scale factor will equal the ratio of the resolution of the virtual screen to the 
resolution of the screen upon which the virtual screen will be displayed. If after a web page is 
displayed, a user zooms in on a portion of the layout, the view window 10210 will correspond to 
only a sup-portion of the virtual screen, and the display scale factor will change from its un-zoomed 
value to reflect the fact that layout elements will be displayed at larger sizes. The current 
embodiment lays out content using the un-zoomed display scale factor to optimize the layout of 
text for display of the entire virtual screen, since it is assumed this is how users will commonly want 
to view the layout. In other embodiments the virtual layout could be optimized for a display scale 
factor other than the un-zoomed display scale factor. 

In the embodiment shown in FIG. 102, this scale factor is stored in the Zoom/Scale Factor 
Control 10216.. In cases where the virtual screen has a resolution of 640 by 480, the view window 
equals the size of the virtual screen, and the view window is displayed on all of a 320 by 240 
display, the display scale factor will be two, meaning that elements are to be drawn on the screen 
of the thin client at 1/2 the pixel resolution at which the browser thinks it is drawing them upon its 
virtual screen. 
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Consideration 10612 of FIG. 106A replaces all font sizes that will be small when displayed 
on the thin client screen with font families that are narrower and taller than the average pixel size of 
the font that would be selected by the consideration 10610 alone. When reducing from a 640 x 
480 virtual screen to a 320 x 240 display screen this can include most or all web page text 
represented in characters, as opposed to bitmap, form. This substitution is done, in part, because 
the subpixel addressable displays used with this embodiment of the invention have three times the 
subpixel resolution in the horizontal direction as they do in the vertical direction. Because of this, 
decreasing the width of characters has a less negative impact on readability than decreasing the 
their height. Thus, to display the maximum amount of relatively easily readable text on such a 
subpixel addressable display screen, this substitution caused the width of characters to effectively 
be scaled down by more than the display scale factor and the height of such characters to 
effectively be scaled down by less than the display scale factor. For example, the fonts of the 
small screen displays shown in FIGS. 56, 57, and 99, 101, 168, 169, 172, 173, and 174 have all 
been substituted by fonts that have been scaled in such a manner. 

The fonts in these figures have a pixel size of eight pixel per em. A majority of the lower 
case letters in this font fit within an advance width of four pixel columns of less. This width of four 
pixel columns or less includes the spacing, if any, that occurs between the shapes of characters 
having such widths. In these particular fonts, over eighty-percent the lower case characters of the 
Roman alphabet fit within such an advance width These characters have an x-height of more than 
four pixel rows, which makes them generally considerably taller than they are wide. As a general 
rule, such a relatively narrow font can represent a larger amount of text within a given area at a 
given level of readability than a wider font. 

The consideration represented by the numerals 10614 and 10616 tests to see if a flag has 
been set to limit minimum font size, indicating that no fonts should be shown on the thin client's 
display below a certain pixel size. Commonly this flag will be set to prevent the display of text that 
is too small to read. It can be unset when the user desires to see a more accurate scaled-down 
representation of how the web page text would normally be laid out if actually shown on a display 
having the virtual screen size. Such a desire is particularly likely when the display scale factor is 
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large, meaning that placing such a minimum limit on text size would drastically alter the 
appearance of the web page's layout. 

If, as is often the case, the system is limiting minimum font size, then steps 10614 and 
10616 prevent the substitute font size from being below a minimum pixel size. In a current 
embodiment of the invention, this minimum pixel size is eight pixels per em. The developers of this 
embodiment developed hinted fonts for subpixel display at seven pixel per em, and although they 
found such fonts relatively easy to read, they received feedback from other users that such small 
fonts were too difficult to bread. 

The limitation on minimum font size often substantially changes the relative size of fonts at 
which a web page's variously sized fonts are actually displayed. 

In some embodiments of the invention, all Web text is displayed at one font size. This 
actually works quite well for most web pages, because in most web pages the truly large fonts are 
represented by bitmaps. 

Once the function 10608 has determined which font family and font size should be 
substitutedifor the font with which the measure string routine has been called, function 10618 
returns the string measurement of the string with which the routine was called, given the size of the 
string's individual characters in the substituted font and font size, after that measurement has been 
scaled up by the un-zoomed display scale factor. 

The return of this value causes the browser's layout engine to lay out the web page using 
font metrics for characters that are scaled, relative to the pixel size at which those characters will 
actually be displays by the un-zoomed display scale factor, which is the ratio of the resolution of 
the virtual screen and the actual resolution at which the virtual screen will be displayed on the thin 
client screen. This means that the virtual screen is being laid out using font metrics that are 
different than the actual font metrics that will be displayed as a result of that layout. 
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If the screen capture and download routine receives a call to a string draw routine 10620 of 
FIG. 106A, this routine causes functions 10621 and 10624 to be performed. 

Function 10621 transforms the screen position at which the string is to start being drawn 
into the corresponding position on the thin client screen at which the string will ultimately be 
displayed. This transformation takes into account the mapping between the view window 10210 
and the virtual screen 10208 illustrated in FIG. 102. This mapping reflects both the current zoom 
setting stored by the control 10216 and a current scroll setting stored by the scroll control 10218 
also shown in FIG. 102. 

Function 10622 tests to see if the substituted font family and size associated with the string 
by the prior call to the measure string routine, described above with regard to numerals 10606 
through 10618, and any other font attributes requested for the display of the current string, are 
different than the current values for such font attributes. The current value for each such font 
attribute is defined by the last value for each such attribute defined by a font commands already 
recorded in the download display list 10212 shown in FIG. 102. If such differences are found, 
function 10623 stores a font command at the current end of the display list changing any such font 
attributes to those appropriate for the display of the current string. 

Function 10624 stores the string with which the string draw routine has been called and 
the transformed screen position just calculated by step 10622 at the end of the download display 
list 10212, illustrated in FIG. 102. As described below with regard FIG. 108, this is done by placing 
a string command in the display list containing the string's transformed start position and its 
characters 

If the screen capture and download routine receives a call to a rectangle draw routine 
10626, this routine causes functions 10628 through 10634 to be performed. Rectangle draw 
commands are commonly called by browsers to create areas of a web page with different 
background color, as well as to draw horizontal and vertical lines that can be used as underlining 
for text or demarcations between different portions of the web page's layout. 
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Function 10628 transforms the geometric values contained in the call to the corresponding 
geometric values with which a corresponding rectangle will be drawn on the thin client's display. 
This includes transforming the rectangle's start screen position, and its width and its height. 

Function 10630 tests to see if the rectangle's color is different than the current (i.e., last) 
rectangle color in the display list. If so, function 10632 adds a background color command to the 
end of the display list changing the current background color to the color specified in the current 
call to the rectangle draw routine. 

Next function 10634 stores the rectangle and its transformed screen position, width, and 
height at the end of the download display list with a rectangle command. 

If the screen capture and download routine receives a call to a bitmap draw routine 10636 
shown in FIG. 106B, this routine causes functions 10638 through 10670 to be performed. Bitmap 
draw routines are called by browsers to display pictures, pictures of fonts, banner ads, and images 
associated with hot zones and other graphical user interface bitmaps of a page. 

In some embodiments, only the first screen of given animations are captured and recorded 
to the download display list to reduce the amount of bandwidth required to display web pages. In 
other embodiments, particularly those with higher bandwidth links such a restriction need not apply 

In the embodiment of the invention that is described with regard to FIGS. 106A through 
106C, bitmap draws associated with certain graphical user interface's are ignored because the thin 
client's programming stores subpixel-optimized, scaled-down bitmaps for such controls. 

Step 10638 tests to see if the URL of the image for which the bitmap draw routine has 
been called is already in a download image list, not shown in the figures, which contains each of 
the images referred to in the download display list. If not, the requested bitmap has not yet been 
processed for the current download and functions 10642 through 10662 need to be performed for 
it. 
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Function 10642 tests to see if the bitmap is a color bitmap. If so it causes functions 10644 
through 10654 to be performed. Function 10644 scans the color images for one or more individual 
areas of sufficient size to justify separate treatment, which each contain only colors from a single 
bicolor spectrum. A bicolor spectrum corresponds to a set of colors that lie in a line in an RGB 
color cube (i.e. a color cube defined by red, green, and blue value ranges in each of its three major 
dimensions). 

For each bicolor portion of the image found that is large enough to justify individual 
processing, function 10646 causes functions 10648 and 10650 to be performed. Function 10648 
performs a bicolor subpixel optimization, of the type described above with regard to FIGS. 42 
through 53, on the current portion of the image using the most extreme ends of its bicolor spectrum 
as its foreground and background colors, and using the current display scale factor to determine 
the extent to which it scales down that portion of the image. This subpixel optimization, like that 
performed in steps 10654 and 10658 described in the next few paragraphs, scales down the image 
by the display scale factor, which is the ratio between the resolution of the image in the virtual 
layout of the proxy browser and the resolution at which it will be displayed on the thin client's 
screen. 

After this subpixel optimization has been performed, function 10650 determines if the 
foreground color is too chromatically unbalanced. That is, it is to close to a pure red, green, or blue 
color. If this is the case, such color purity would decrease the accuracy with which it can display 
the spatial resolution of the color image. If this is the case, the foreground color can be replaced 
by a corresponding color that is closer to a grayscale value, and thus which will allow more 
accurate spatial representation. 

In some embodiments of the invention such foreground color substitution will not be used 
because it might upset the color balance of the color image. In general it is best not to use such 
foreground color substitution unless the foreground color appears throughout a substantial portion 
of the entire color image. Such color substitutions would be best where small fonts or other shapes 
are being represented by the bitmap image being optimized. In other embodiments of the 
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invention the background color associated with a bicolor image could be changed. But the 
Changing of the background colors of images on web pages is often unadvisable. 

For each non-bicolored portion of the current image, function 10652 causes step 10654 to 
perform a multicolored subpixel optimization, of the type described above with regard to FIGS. 14 
through 41 , on that portion of the bitmap at the current display factor. 

If the bitmap for which the bitmap draw routine has been called is a grayscale bitmap, 
function 10656 causes step 10658 to perform a bicolor subpixel optimization on the bitmap using 
black and white as the foreground and background colors at the current display scale factor. 

Then function 10662 stores the scaled-down, subpixel-optimized bitmap at the end of the 
image list with a unique image ID, its URL, and its scaled width and height. 

Whether or not the image with which the bitmap draw routine has been called was 
previously in the image list, by the time the program advances to function 10664 it will be in that 
list, and will have been assigned an ID number and a transformed width and height. At this time 
function 10664 transforms the screen position with which the bitmap draw routine has been called 
for the image to one applicable to the thin client's screen, and then stores an image location 
command of the type shown in FIG. 108 having the image's image ID, its transformed screen 
position, and its transformed width and height at the end of the download display list. 

•In some embodiments of the invention all bitmap images are subpixel-optimized using the 
multicolor subpixel optimization routine. In other embodiments only grayscale bitmaps undergo 
any bicolor subpixel optimization. 

In some embodiments of the invention vector images can be handled by performing 
subpixel optimization upon the shapes defined by such vector descriptions. In some such 
embodiments such subpixel optimization is performed on the proxy, but in others it is performed on 
the thin client. One of the advantages of vector, or geometrically defined, drawings it is the 
compactness with which their descriptions can represent an image. Thus, when bandwidth to the 
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thin client is a primary restriction, it might well make sense to download vector descriptions of 
images and have the thin client then render them using subpixel optimization. 

It is possible in some embodiments, to have image recognition performed upon images, 
and then have the recognized images downloaded to the thin client in a symbolic representation. 
For example, it is common in many web pages to represent large text with bitmaps. Optical 
character recognition could be performed on such bitmaps, and corresponding characters and their 
font, or an approximation of their font could be downloaded symbolically, so as to reduce the 
bandwidth required in order to describe the page to the thin client. 

If the screen capture and download routine receives a call to the routine to create a control 
object, such as a radio button, check box, text field, or button from the browser, the controlCreate 
routine 10666 shown in FIG. 106C causes functions 10667 through 10670 to be performed. 
Function 10667 transforms the screen position at which the browser has requested a control to be 
drawn to the location at which it is to be drawn in the thin client's screen. A function 10668 places 
a corresponding control create command, as indicated in FIG. 108, in the download display list, 
including its corresponding text label, and function 10670 creates a corresponding browser side 
portion of the control object. 

In this embodiment of the invention the functionality of a control object shown in the thin 
client's screen is shared between the proxy and the thin client. State information, such as whether 
not a check box has been checked, or which set of radio buttons has been pushed, is stored on the 
thin client. This prevents the need for communication from the thin client to the proxy every time 
the user enters information into such a control object. Usually it is only when the user clicks some 
sort of a button that indicates the information stored for such controls is to be transmitted to the 
remote server computer that originally generated the web page, that the client needs to send such 
information to the proxy, for relay to such a server. 

In other embodiments of the invention having a higher bandwidth link to the thin client, it 
might be desirable to simplify the code of the thin client, by having more or substantially all of the 
functionality associated with individual control objects run on the proxy. 
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When the screen capture and download routine determines that the screen redraw 
requested by function 10602 of FIG. 106A is complete, function 10672 of FIG. 106C causes 
function 10674 to call the download display list routine 10700 shown in FIG. 107. 

As shown in FIG. 107, the download display list routine has a function 10702 that places 
all elements in the download display list that are to be totally or partially newly displayed on the 
new thin client's screen in a download stream. Normally this includes any elements in the 
browser's virtual screen that occur within the current view window. As is explained below, 
however, in the case of a scroll in which a significant portion of the prior bitmap on the thin client's 
screen can be reused, only elements that occur at least partially in the portion of the view window 
that is not in the reusable portion of the thin client screen's current bitmap are placed in the 
download stream. 

In many embodiments of the invention the functions of FIGS. 106A through 106C that 
creates the download display list do not enter an element on the download display list if it does not 
fit within the view window. In other embodiments this filtering takes place in function 10702. 

In some embodiments of the invention elements that are downloaded are clipped, so that 
only those portions of such elements that are to actually fit within the thin client screen are 
downloaded. This would have the benefit of decreasing the number of bits required for download, 
but it would add computational complexity. 

Once all the elements on the download display list to be shown on the thin client screen 
have been placed in the download stream, function 10704 places the bitmaps of all images with a 
corresponding image location command in the download stream at the end of the download 
stream, as indicated by the numeral entries 10818 in FIG. 108. Some embodiments of the 
invention, before they places such bitmaps at the end of the download stream perform a lossy 
compression on them. In some embodiments, the algorithm used is one that clusters the color 
values in the image into clusters of colors having visually imperceptible differences in RGB color 
values, using a metric that takes into account the fact that green color values differences are more 
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perceptible than red color value differences, and that red color value differences are more 
perceptible than blue color value differences. 

Then function 10705 compresses the download stream, including the images previously 
compressed by the lossy algorithm, using a lossless compression algorithm. Standard prior art 
lossless compression algorithms can be used for this purpose. 

FIG. 108 is a schematic illustration of such a download display stream. In some 
embodiments such a stream is actually represented using a markup language. 

The font commands 10812 shown in FIG. 108 represent font commands recorded in the 
display list by function 10623 of FIG. 106A. 

The string commands 10814 of FIG. 108 represent commands recorded in the download 
display list by the step 10624 of FIG. 106A. 

The background color commands 10806 of FIG. 108 represent the background color 
commands entered by the function 10632 shown in FIG. 106A. 

The rectangle commands 10808 of FIG. 108 represent rectangle information stored by 
function 10634 of FIG. 106A. 

The image location commands 10810 shown in FIG. 108 represent image location 
commands recorded by the function 10664 of FIG. 106B. 

The control commands 10816 of FIG. 108 represent control commands placed in the 
download display list by the function 10668 of FIG. 106C. 

Returning now to FIG. 107, once all the elements for the download stream have been 
selected and the stream is ready to be sent, function 10706 opens a socket connection between 
the browser computer and the thin client, and then function 10708 sends the download stream's 
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display list information down to the thin client. The thin client then displays information, as is 
described below in greater detail with regard to FIGS. 109A through 109C. 

Returning now to FIG. 106C, once the call in the function 10674 to the download display 
list routine is complete the function 10676 clears the display list, so the new display list can be 
created for the next screen that is to be downloaded to the thin client. 

Returning now to FIG. 105A, we have just described the completion of the screen capture 
and download routine called by function 10510 shown in that figure. 

As shown in FIG. 105A, if the browser's proxy code receives a query from another portion 
of the browser code for the state of one or more control objects displayed on the thin client's 
screen, function 10514 causes functions 10516 and 10518 to be performed. Function 10516 
sends a query to the thin client for the state of that one or more control objects. When such state 
information is received from the thin client, function 10518 returns it to the programming that made 
the request for such state information. 

As was described above with regard to functions 10666 through 10670 of FIG. 106C, this 
embodiment of the invention actually has the thin client draw and store state information about 
individual control objects, such as radio buttons, check boxes, and text entry fields, to reduce 
communication bandwidth as the user changes information prior to selecting to have such state 
information submitted to the web site on whose web page such controls are shown. Commonly 
when the user clicks on a submit button the associated click event is transmitted up to the proxy 
computer, it has its screen coordinates transformed the corresponding coordinates on the virtual . 
layout screen, and then it is placed in the browser's event queue for the browser code to respond 
to that click event as if it had been generated on the screen, having the virtual screens resolution, 
that the browser thinks it is displaying. Once this is done, the browser code traditionally asks for 
the state of all of the current web page's control objects, so it can post that information back to the 
web server from which the current web page came. It is such requests that cause the operation of 
functions 10514 through 10518. 
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If the browser's proxy code receives a scroll or move command from the thin client, 
function 10520 causes functions 10522 through 10534 of FIG. 105A are performed. 

Function 10522 moves the view window 10210 shown in FIG. 102 relative to the browser's 
layout 10206 in response to the scroll or move. Then function 10526 tests to see if any significant 
portion of the view window that was in the view window before the move is still in the view window 
after the move. If this is the case, it means a substantial portion of the bitmap currently being 
displayed on the thin browser screen can be reused in the display after the requested scroll or 
move is accomplished. In this case, function 10528 places a scroll command 10804, illustrated 
near the top of the download stream in FIG. 108, at the start of the new display list that is to be 
created for the scrolled screen. Such a scroll commands includes an XY shift value that indicates 
which portion of the thin client's prior screen bitmap is to be reused. 

In FIG. 108 both a clear command 10802 and a scroll command 10804 are shown at the 
start of the download stream, so that both can be illustrated. In the current embodiment only one 
of these two commands, the clear command or the scroll command will start a given download 
stream, with the first being used if the screen of the thin client is to be totally redrawn, and the 
second being used if a portion of the thin client screen's prior bitmap is to be shifted for reuse in the 
new screen. 

The reuse of a substantial portion of a screen display that has been previously 
downloaded and drawn, made possible by the use of the scroll command, can substantially reduce 
the amount of data that has to be downloaded to the thin client in scrolls that involved relatively 
small changes in position. This can substantially speedup the rate at which scrolled screens can 
be displayed on the thin client, particular in situations in which there is a limited bandwidth between 
the browser and the thin client, such as if they're communicating over the relatively slow digital 
cellular link common at the time this application is being filed. 

If the moved view window that results from a scroll or move command includes a portion of 
the web page's layout not currently in the virtual screen 10206, shown schematically in FIG. 102, 
function 10530 of FIG. 105A causes functions 10532 and 10534 to be performed. Function 10532 
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scrolls the browser's virtual screen so that all of the view window will be contained within it, and 
then function 10534 requests a redraw from the browser for the newly moved virtual screen. Once 
this redraw is complete functions 10506 and 10510 will capture the newly drawn elements and will 
draw them, as has been described above with regard to FIGS. 106A through 106C. 

If the browser's proxy receives a zoom command from the thin client, function 10536 of 
FIG. 105A causes functions 10538 through 10552 to be performed. 

Function 10538 changes the display scale factor according to the zoom change. 

Function 10540 scales the view window relative to the browser's virtual window according 
to the selected zoom. 

Function 10542 checks to see if the scaled view window includes portions of the web 
page's layout not currently contained within the virtual screen. If so, it causes function 10544 to 
scroll the virtual screen or change its resolution to make the scale view window fit within the virtual 
screen. 

If scrolling the virtual screen will enable the new view window to fit within the virtual screen, 
there is no need to re-layout the web page, and the zoom can be used to display the same layout 
as existed before the zoom, by showing a different location within it and/or by displaying it at a 
different scale factor. If, however, the zoom is a zoom out that causes the view window to be 
larger than the virtual screen size, in the embodiment shown in FIG. 105A, this will require that the 
web page be laid out at a new virtual screen size that allows the view window to fit entirely within 
the virtual screen, so that the proxy browser can handle any input supplied to any portion of the 
view window displayed on the client as if it had occurred at a corresponding location on the proxy 
browser's virtual screen. In the embodiment being described, this may cause the web page to be 
displayed at a new layout if the new virtual screen resolution is larger than the layout resolution 
used in the previous layout, and this can cause line breaks to occur in different locations. 
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In other embodiments of the invention, such as ones in which proxy browser was 
designed, rather than patched, to support zoomed views, and such as the ones described with 
regard to FIG. 1 15 in which the client zooms directly relative to a download of an entire layout, 
extreme zoom outs need not require a re-layout of the web page. 

Finally function 10552 calls for a screen redraw. This causes the screen capture and download 
routine to capture the redraw of the current view window with the new zoom scale factor, and 
download corresponding display information to the thin client so it can display the web page at the 
new zoom setting. 

As indicated in FIG. 105B, if the browser's proxy receives a virtual resolution change 
command from the thin client, function 10554 causes functions 10556 through 10560 to be 
performed. Function 10556 changes the browser's virtual screen resolution, stored in the virtual 
resolution control 10214 shown in FIG. 102, to the requested resolution. Then step 10560 calls for 
a screen redraw. This is because the browser re-lays out the current web page at the new virtual 
screen resolution, and redraws all of the virtual screen at the display's scale factor corresponding 
to the ratio between the number of pixels the view window has in the virtual screen relative to the 
number of pixels it has on the thin client screen. 

Such a change in virtual resolution changes the size at which a layout is performed relative 
to the size of images and text within such a layout. Such a change in relative layout size changes 
the size at which images and text will be displayed on the screen, unless the user makes a change 
in the relative size of the view window relative to the virtual screen that cancels such a change in 
size. In the absence of such a compensating change in relative view window size, decreasing the 
virtual resolution, increases the size at which images and text will be shown on the screen, and 
tends to make the text lines shorter relative to the size of the fonts shown on them, so as to allow 
more of text lines to fit on the screen at one time at a larger text size. Thus, changes in virtual 
layout size can be used to provide a certain type of zoom capability to the display of web pages. 

The inventors have found that quite good readability can be supplied using virtual screen 
of 640 by 480 when displaying web pages on a typical PDA-sized 320 by 240 screen, which 
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involves scaling down the layout by a factor of 2. However, the invention can be used to display 
web pages at even more reduced scales, such as displaying an 800 by 600 virtual screen 
resolution on a PDA-sized 320 by 240 display, even though readability will suffer, so as to enable a 
user to see how the web page might look when laid out for larger resolution displays. Of course, if 
the reader chooses to have the minimum font size limited, as was described above with regard to 
function 10614 of FIG. 106A, the text, even with such a large virtual resolution would still be shown 
with readable fonts, although the layout of the page would be quite different than that originally 
intended for display at such a resolution, because of the relative increase in font size that would 
result. 

As indicated by function 10562 of FIG. 105B, if the browser's proxy code receives other 
user input from the thin client associated with a click on the thin client's screen, function 10564 
transforms the thin client screen position associated with the click to the corresponding position on 
the virtual screen, and function 10566 relays the event to the browser's event queue so that it can 
respond to it as if the user had actually clicked on the virtual screen that most of the browser's code 
thinks it is laying out. 

This is the method by which the browser on the proxy responds to input from the user of 
the thin client to select web links, whether they be text links or image links, on the web page 
displayed on the thin client. For example, if the user clicks on a link displayed on the thin clients 
screen, the corresponding click will be relayed to the browser on the proxy, which will act as if the 
user had clicked on the same link in the virtual screen that it thinks it is displaying. The proxy's 
browser then responds by issuing an HTTP request over the Internet corresponding to the link. 
When the web page corresponding to that link is received, the browser will lay out and seek to 
display it on the virtual screen, causing functions 10506 and 10510 of FIG. 105A to capture the 
information contained in that portion of the layout corresponding to the view window and to 
download it to the thin client for display on it's screen. As a result, the user of the thin client is able 
to surf the Web, in much the same manner as a user of a normal browsing computer. 
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FIGS. 109A through 109C are highly simplified pseudocode representations of code 10900 
on the thin client computer designed to help it operate in conjunction with the proxy browser to 
enable its users to browse the World Wide Web using its screen. 

Function 10902 of FIG. 109A responds to the receipt of all or an initial portion of the 
download stream sent to the thin client by function 10708 of FIG. 107. It does so by starting to 
respond to the individual commands, of the type illustrated in FIG. 108, contained in that stream in 
the order in which they are received. It starts doing this as soon as one or more such commands 
are received so that the work of drawing the new screen need not be delayed until the download 
stream has been fully received. The response to each different type of command contained in the 
download stream is indicated by the functions numbered 10904 through 10956 in FIGS 109A 
through 109B. 

As indicated by functions 10904 and 10906, when the thin client reads a clear command in 
the download stream it causes the bitmap displayed on its screen to be cleared, such as by being 
set to a totally white value. 

When the thin client reads a scroll command in the download stream, function 10908 
causes functions 10910 and 10912 to be performed Function 10910 copies the portion of the thin 
client screen's bitmap that is to be reused after the scroll specified in the scroll command to a new 
position on that screen indicated by the XY shift value included in the command. Then function 
10912 clears the remaining portion of the screen. 

When the thin client reads a background color command in the download stream, functions 
10914 and 10916 set the current rectangle background color variable to the color specified in the 
command. This causes all rectangles drawn by the thin client in response to rectangle commands 
until the background color value is changed to have that specified color value. 

When the thin client reads a rectangle command in the download stream, functions 10918 
and 10920 draw a rectangle, using the current background color, having a screen position, width, 
and height specified in the command. 
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When the thin client reads an image locations command, functions 10922 and 10923 do 
nothing at that time. This is because the bitmap's necessary to draw the image referenced in such 
an image locations command usually will not have been received at such time. In other 
embodiments, the browser associates rectangle draw commands with images, which will cause the 
portion of the thin browser screen associated with images to have a rectangle drawn on them 
indicating where a bitmap image is to be displayed. 

When the thin client reads a font command, functions 10924 and 10926 set the value of all 
font attributes listed in the font command to the values listed for those attributes in that command. 
In different embodiments of the invention different font attributes can be used. It is preferred that at 
least font family, font size, and font foreground color be supported font attributes. 

When the thin client reads a string command in the download stream, function 10928 
causes functions 10930 through 10940 to be performed. 

Functions 10930 tests to see if the thin client has in its font bitmap cache a bitmap for each 
character of the current string in the current size and font family specified by the current font 
attribute values. If not, functions 10932 through 10936 are performed. 

Function 10932 sends an HTTP request over the thin client's Internet connection to the 
font server 134 described above with regard to FIG. 2. When the requested font is received from 
the font server, functions 10934 and 10936 place it in the thin client's font bitmap cash. 

It should be noted that some embodiments of the invention permanently store, as part of 
the thin client browser software, a sufficient set of font bitmaps so that the use of the functions 
10930 through 10936 are not necessary. In other embodiments, subpixel-optimized font outlines 
are either stored permanently by the thin client or are requested as needed, as are the font bitmaps 
in the example described in FIG. 109A. 
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When the thin client has all of the font bitmaps necessary to render the current string, 
functions 10938 and 10940 draw the string using the current font attribute values including 
foreground color, upon the screen at the specified screen position. In the current embodiment font 
bitmaps are represented as alpha value bitmaps of the type described above with regard to FIGS. 
60, 96, and 97. When doing so, the background color is drawn from the portion of the screen 
bitmap over which the string is to be drawn. 

In some embodiments, in order to reduce computation, the color value of the portion of the 
screen over which the string is to be drawn is sampled at a relatively few number of points, and the 
average of those sampled color values is used as the background color for the entire string display, 
as is described above with regard to FIG. 97. 

In the embodiment being described, all of the strings contained in the download stream are 
single line text strings, many of which may have resulted from the wrapping of continuous text 
across line boundaries by the proxy browser's layout engine. As a result, in this embodiment, the 
thin client does not have to perform any such wrapping of text. 

Function 10940 draws a bitmap image of a strings by composing it from a plurality of 
separate font bitmaps corresponding to the letters of the string. Normally in such composition each 
different separate character will be represented by a different separate font bitmap. 

It is preferred that the fonts used in such composition at different font sizes (such as 
different font sizes caused by changes in the display scale factor) have the shape and pixel 
alignment of each character selected to improve readability at each such font size. In most 
embodiments this improved readability is produced by selecting the character shape and position 
relative to a font bitmap so as to increase the alignment of the character shape with the pixilation of 
the bitmap. Such shape and pixel alignment is particularly critical when dealing with font bitmaps 
of ten pixels per em or less, and is even more critical at eight pixels per em or less. This is 
because as font bitmaps became smaller they become more difficult to read because of their more 
course pixilation, and thus it becomes even more critical that they have character shapes and 
alignments selected to fit such pixelation. 
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In many embodiments of the invention the font bitmaps used by step 10940 at smaller 
scales are subpixel optimized bitmaps created by non-linear color balancing of the type described 
above, in which only color imbalances that occur within a pixel are distributed. When such 
subpixel-optimization is combined with outlines that have been properly shaped and aligned to 
better match their coarse bitmap pixelation, the resulting bitmaps drawn are amazing easy to read 
considering their small pixel size. In other embodiments of the invention non-subpixel-optimized 
anti-aliased fonts are used that similarly have outlines shaped and aligned to improve readability at 
such small pixel sizes. 

Turning now to FIG. 109B, when the thin client reads a control command from the 
download stream, function 10942 causes functions 10944 through 10948 to be performed. 

Function 10944 tests to see if it already created a data or program object corresponding to 
the control ID specified in the current control command. If not function 10946 creates such a data 
or program object of the type specified in the control command and associates it with the control ID 
specified in that command. 

Then step 10948 draws a subpixel-optimized bitmap of the specified type of control object 
on the thin client's screen at the location specified in the control command. It then draws the text 
associated with the control on the control object's bitmap using subpixel-optimized fonts. Then it 
associates a hot zone, having a display screen position corresponding to the control's bitmap, with 
the data object or program object representing the control on the thin client. 

When the thin client reads an image command from the download stream, function 10950 
causes functions 10952 through 10956 to be performed. 

Function 10952 scans the current display stream for all occurrences of an image location 
command that has the same image ID as the current image command. For each such image 
location command, it causes function 10954 to draw the bitmap at the location specified by that 
image location command upon the thin client's screen. As with all the thin client's draw functions, 
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any portion of the image that does not fit on the thin client screen is clipped in such draw 
operations. 

Next function 10956 redraws all other items in the display list that occur at the same 
location as any of these drawn bitmaps. This is necessary because it is common for web pages to 
place text on top of images, and, thus, it is desirable that any strings that are intended to be 
displayed at the same location as a bitmap image be redrawn after those images are drawn. In 
one embodiment of the invention, the thin client merely redraws all non-image elements of the 
download stream's display list that occurs after the first image location command in that list. 

If the user clicks on a hot zone 1 1000 associated with a text entry field, as indicated in FIG. 
110, functions 10958 and 10960 of FIG. 109B cause a keyboard routine comprised of functions 
10962 through 10978 to be executed. 

Function 10962 displays a pop-up user keyboard 1 1 102 and text edit field 1 1 104, 
illustrated in FIG. 1 1 1 , on the thin client's screen. Then a loop 10964 is performed until the user 
presses the enter key on the pop-up keyboard. During this loop each time a user types a text 
character, function 10966 causes function 10968 to place a subpixel-optimized text bitmap of the 
character on the pop-up keyboard's text edit line at the current cursor position and moves the 
bitmap of the cursor to a position after the newly drawn character, and then function 10970 adds 
the typed character to a temporary text edit string associated with the pop-up keyboard's 
programming. 

When the user presses the enter key of the pop-up keyboard, function 10972 causes 
functions 10974 through 10978 to be performed. Function 10974 stores the value of the temporary 
text edit string associated with the pop-up keyboard in the text edit control for which the pop-up 
keyboard has been evoked. Then function 10976 draws the characters of that text edit string, 
using subpixel optimized bitmaps, in the bitmap of the text entry field 1 1000 of the control object on 
the thin client's screen, as shown in FIG. 112. 
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Then function 10978 removes the pop-up keyboard from the thin client's screen by 
drawing over it the bitmap that was displayed on-screen before the pop-up keyboard was drawn. 

FIG. 1 13 illustrates that the pop-up keyboard routine can be used for other purposes 
besides entering text in text entry field. Although it is not represented in the pseudocode of FIGS. 
109A through 109C, the pop-up keyboard can also be used to enter the URLs of web pages a user 
would like to see displayed on the thin client. 

FIG. 1 14 is virtually identical to FIG. 1 13, except it illustrates an embodiment of the 
invention that has a button bar, or Toolbar, at the top of its graphical user interface. This button bar 
includes, at its leftmost end, "back" and "forward" buttons of the type commonly found in Web 
browsers. It also includes buttons labeled "R", "B", and "H", which correspond to a refresh button, 
a bookmark button, and a history button, respectively, which are also functions commonly found on 
Web browsers. The button bar also includes an URL text entry field, which if clicked will cause the 
pop-up keyboard shown in FIG. 1 14 to appear. When the pop-up keyboard is not being displayed, 
this text entry field displays the URL of the current web page displayed on the thin client's screen. 
In one embodiment of the invention a user can select whether or not to display such a toolbar by 
pressing a hardware button. In this embodiment, even when such a tool bar is not shown the user 
can use hardware buttons to invoke some of the more common web browsing functions, such as 
the back command and forward commands. 

In other embodiments of the invention, such a graphical user interface Toolbar would 
preferably also include buttons or menus allowing the user to access other functionality of the 
browser, including changing the zoom and/or relative layout size (i.e., virtual layout resolution) of a 
web page's display. 

Returning now to FIG. 109B, if the user clicks on a hot zone of a button or menu item 
control, function 10980 causes functions 10981 and 10982 to be performed. 

Function 10981 changes the appearance of the button or menu item appropriately. In the 
case of a button, the bitmap associated with the button is redrawn to indicate the button is being 
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pressed. In the case of a menu item, either a submenu will be displayed, or the display of the 
menu item will be removed, depending upon whether or not a final selection has been made. 

If a final selection has been made in the case of a menu item, or the button has been 
pressed and released, function 10982 sends the button's or menu item's control ID and an 
indication that it has been selected up to the browser, which responds by causing the 
corresponding button or menu item control object on the browser to act as if it had been clicked. 

If the user clicks on the hot zone associated with another type of thin client control, 
function 10983 changes the appearance of the control's bitmap on the thin client's display 
accordingly. For example, in the case of a check box, a check would either be displayed or 
removed from the display of the control on screen. Then step 10985 stores the corresponding 
state change in association with the control object. As stated above, in the embodiment being 
described, the state of such control objects are not communicated to the browser until the browser 
requests such information, in order to reduce communication demands. 

If the user clicks on any other portion of the thin client's screen not associated with the 
control interface of the thin client program or its computer, functions 10986 and 10987 send an 
event corresponding to that click up to the proxy browser along with the screen location at which it 
occurred. As was described above with regard functions 10562 through 10556 of FIG. 105B, the 
browser will transform the location of such a click to the corresponding location on its virtual 
screen, and will respond to such a click as if it occurred upon the screen that the browser thinks it 
is drawing at the resolution of the virtual screen. In some embodiments, to further reduced 
communication demands, the thin client will only report such other clicks to the browser if it has 
reason to believe they corresponds to a user input the proxy's browser is supposed to respond to. 

Referring now to FIG. 109C, if the thin client receives a query from the proxy browser 
asking the state of one or more control objects, function 10988 causes function 10989 to query the 
state of the corresponding controls on the thin client, and function 10909 to transmit that state 
information to the proxy browser. As was described above with regard to function 10518 of FIG. 
105A, the proxy browser will then return such requested information to the part of the browser that 
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requested it, as if that information were part of the current state of the corresponding control 
objects associated with the virtual screen. 

If the user of the thin client enters the command to scroll its screen, functions 10991 and 
10992 upload that scroll command to the proxy. This causes the functions 10520 through 10534 
described above with regard to FIG. 105A to generate and download a new download stream for 
the display of the current web page at the newly scrolled position. 

If the user enters the command to change the zoom, i.e., scale, of the image displayed on 
the thin client, functions 10993 and 10994 upload a corresponding zoom command to the proxy. 
This causes the functions 10536 and 10552 described above with regard FIG. 105A to cause a 
new download stream to be downloaded to the thin client for display of the current web page at the 
new zoom setting. 

If the user enters the command to change the virtual resolution of the thin client's display, 
that is to change the resolution at which the virtual screen on the proxy browser is laid out, 
functions 10995 and 10996 upload the selected virtual resolution to the proxy. This causes 
functions 10554 through 10560, described above with regard FIG. 105B, to change the proxy 
browser's virtual screen resolution and to have the virtual screen re-laid out at the new resolution 
and a corresponding downloads stream to be sent to the thin client, so it can display the portion of 
the virtual screen corresponding to the window at the current zoom setting upon the thin clients 
screen. Since a change in the virtual resolution tends to change the relative size of layout 
elements such as text and images relative to the width of the virtual layout, the virtual resolution is 
sometimes referred to as the relative layout size in this specification. 

As indicated at the bottom of FIG. 109C, if the user enters another command associated 
with the thin client's control graphical user interface, function 10997 will cause a correspondingly 
appropriate response, indicated by the ellipses 10999 to be performed. Such other functions can 
include the selection of bookmarks, the accessing of bookmarked web pages, back and forward 
functions, or any other function that can be part of a browser's user interface. Such demands can 
be selected by use of physical buttons or other physical inputs on the thin client computer, by the 
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selection of graphical objects, such as buttons, menu item, or dialog box controls, or virtually any 
other known graphical user interface technique. 

Figures 1 1 5 through 117 relate to an alternate method for enabling a thin client computer 
to browse the web through a proxy server. In this embodiment the entire layout 10206 of a web 
page created by the proxy computer is downloaded to, and cached by, the thin client, as indicated 
in FIG. 117. As will be described, this allows the thin client to scroll within the layout at 
substantially higher speeds, although it can increase the total number of bits downloaded, since it 
attempts to download the entire layout and all images for each web page viewed. 

FIG. 115 illustrates portions of the proxy browser's code 1 1500 that can be used with such 
a page layout caching scheme. 

In this embodiment of the invention if the proxy browser receives a request for a web page 
from the thin client, functions 1 1 502 causes functions 1 1 504 through 1 1 524 to be performed. 

As indicated by the pseudocode associated with function 1 1502 in FIG. 1 15, in this 
particular web caching embodiment the thin client can request a web page with a desired view 
setting for that page, including a desired virtual resolution, zoom setting, and view window position. 
This is done to allow a user to associate such view settings with a bookmark, including a particular 
URLs or a portions of a URL path name, so as to allow the user to automatically see such web 
pages at a desired virtual resolution, zoom setting, and view window position, without having to 
separately enter such setting values each time the page is requested. This, for example, would 
allow a user view commonly accessed web pages with the display automatically zoomed in on a 
desired portion of that page. 

Once a request for a web page has been received from the thin client, function 1 1504 on 
the proxy browser requests that web page from the server identified in the URL of the request from 
the thin client. When the web page is received from the server, function 1 1506 causes functions 
1 1 507 through 1 1 51 6 to be performed. 
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Function 1 1507 causes the layout engine of the browser on the proxy to lay out the 
received web page at the virtual screen resolution associated with the view setting specified in the 
request from the thin client for the web page. This layout is made using scaled string 
measurements for substituted fonts, in a manner similar to that described above with regard to 
functions 10606 through 10618 of FIG. 106A. The scale factor used is determined by the view 
setting specified in request for the current page. 

Function 1 1508 specifies a virtual screen position relative to the resulting layout that will 
include the view window implicit in the view setting of the current request. Thus for example, if the 
view setting requests to see the rightmost 640 by 480 portion of the layout, and the layout is forced 
to have a width of 800 pixels, the virtual screen position would extend from approximately pixel 
column 160 over to pixel column 800 in the layout. 

Function 1 1518 causes functions 1 1520 to scale and subpixel-optimize each image 1 1702, 
illustrated schematically in FIG. 117, received in association with the web page being laid out. 

Once all the images referenced in the web page have been received, scaled, and subpixel 
optimized, function 1 1522 causes function 1 1523 to create a display list for that layout, and to 
compress that display list and all its associated subpixel-optimized, scaled-down images. Then 
function 1 1524 transmits that compressed data to the thin client in a download stream that includes 
the web page's layout, followed by its scaled-down, subpixel-optimized images. 

In other embodiments of this aspect of the invention, portions of the layout could be 
downloaded to the thin client before all or even any of the images have been downloaded to the 
proxy server. This would allow the thin client to start displaying the text of the web page faster. 
Since many web pages contain a description of the size of all their images, often a proper layout of 
a web page can be performed before images are received by the software performing such a 
layout. Where this is not the case, the page will have to be re-laid out when it learns the size of all 
the images, and this new layout could then be sent down to the client. 
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If the user receives a request from a thin client to rescale and subpixel-optimized one or 
more images previously downloaded at a different scale, function 1 1526 through 1 1532, rescale 
and subpixel-optimize, compress, and download such images to the thin client. This allows the 
user to view the web page at a different subpixel optimized size if he or she seeks to view the 
downloaded web page layout at a different zoom setting. 

If a screen input's event is received from the thin client function 1 1534 causes functions 
1 1536 through 1 1542 to be performed. 

Function 1 1536 tests to see if the page layout coordinates associated with the command 
corresponds to a portion 10206A of the web page layout 10206 currently mapped to the proxy 
browser's virtual screen 10208, shown in FIG. 117. If not, function 1 1538 scrolls the virtual screen 
to a new portion 1 0206B of the layout that includes the layout coordinates associated with the 
command. 

Function 1 1540 calculates the virtual screen coordinate corresponding to the page layout 
coordinate of the received screen event. Then function 1 1542 places the input screen event with 
its virtual screen coordinates in the browser's event queue, so that it can respond to that event, 
such as the clicking of a link, as if the user had clicked at its corresponding virtual screen 
coordinates on the virtual screen itself. 

FIG. 1 16 is a highly simplified pseudo code description of portions of the thin client code 
that can be used to support the page layout caching scheme illustrated in FIGS. 1 15 and 1 17. 

If the thin client starts to receive a download stream containing a page layout's display list, 
function 1 1602 causes function 1 1604 and 1 1606 to be performed. 

Function 1 1604 sets the mapping of the view window (such as the view window 10210A 
shown in FIG. 117) relative to the page layout, and then calculates the current client-side display 
scale factor based on that mapping. 
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Function 1 1620 displays any portion of the downloaded page layout that falls within the 
current view window, using the current client-side scale factor. This process includes the functions 
11622 through 11630. 

Function 1 1622 displays each string element that occurs within the current view window 
with font sizes that are a function of the current client-side scale factor. When it does so it adjusts, 
if necessary, for any disproportionate changes in the relative size of characters that might result 
from the uneven effects of font hinting as the pixel size at which such characters are displayed 
changes. It does this by using techniques for compensating for the discontinuities and 
disproportionalities, such as changing spacing between characters, similar to those traditionally 
used to provide a WYSIWYG correspondence between the display of text on a computer screen 
and its appearance when printed at a much higher resolution. If font bitmaps having a different 
size that than previously displayed are required, font bitmaps for such differently sized characters 
can be either accessed from storage on the thin client, accessed from a network font server, or 
rasterized at the needed size from a font outline. 

If the display created by function 1 1620 is at a different scale factor than that for which the 
bitmap images 10818 have been scaled-down, function 1 1624 causes functions 1 1626 through 
1 1630 to be performed. These request the proxy server to rescale at the new scale factor and 
subpixel-optimize all images that are totally or partially within the view window. Then the bitmaps 
of the same images are locally rescaled from the formerly downscaled and subpixel-optimized 
images 10818 stored on the thin client and displayed on the thin client screen to provide a 
temporary representation for such images. Then, when the requested images that have been 
rescaled from the original, higher resolution bitmaps associated with the web page have been 
received by the thin client from the proxy server, they are drawn at the appropriate location on the 
display screen. 

In some embodiments, when a user changes the zoom of the display, the bitmaps of any 
images corresponding to a portion of the page on the screen at the new scale factor are displayed 
with a quick, but crude representation of the image generated on the thin client to provide the user 
with a temporary representation of such images until the properly subpixel-optimized versions of 
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the images have been downloaded. Such quick representations are relatively simple to generate 
when the new scale factor is an integral ratio of the scale factor of the previously downloaded 
bitmaps. When this is not the case, the temporary representation could be produced in any of a 
number of ways. For example, they could be subpixel optimized images generated by one of the 
methods described above with regard to FIGS. 12 through 53. The basic procedures described in 
these methods can be applied even when the source image is of a lower resolution than the output 
image. An even more computationally simple method would be to display the prior images as 
images having integral scaling ratios and that have been either cropped or scaled to a size equal to 
or smaller than the proper scale so to not take up more space than the properly scaled images that 
are intended to be drawn over them. 

If the user generates a screen input to be sent to the proxy browser, function 1 1632 
through 1 1636 transform the thin client screen coordinates of the input to a corresponding page 
layout coordinate. Then the screen input and corresponding page layout coordinate are uploaded 
to the proxy browser. The proxy browser then responds to such an input using the functions 1 1534 
through 1 1542 described above with regard FIG. 115. This causes the proxy browser to respond 
to such screen input as if the user had clicked on a corresponding portion of the web page on the 
proxy browser's virtual screen. 

Caching schemes, such as that just described with regard FIGS. 115 through 117, that 
allow the thin client to store more than the portion of a web page currently displayed on a screen, 
can be used to allow a user to scroll and/or zoom more rapidly relative to web page's content. This 
is particularly true if the thin client has a relatively low bandwidth to its proxy server. 

The embodiment of such a caching scheme that has just been described can be made to 
allow relatively quick browsing of the web even with bandwidths as low as those associated with 
current digital cellular communication rates commonly available at the time of this application. This 
is because all the content, except images, included in most web pages can normally be 
compressed to fit into 3,000 bytes or less. Thus, at current digital cellular communication rates, the 
entire text portion of most web pages could be downloaded in several seconds, and the initial 
portions of it could be drawn in even less time. Of course the downloading of the images might 
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take more time, but all but large images at the start of the web page would commonly be displayed 
within a few seconds. And with faster communication links this delay can be reduced 
tremendously. 

FIGS. 118 through 120 illustrate aspects of the invention that can be used in virtually any 
Web browsing environment, but which are particularly useful when browsing the Web on small 
screens. This includes use on small screen devices such as the thin client computers discussed 
above. Because these aspects of the invention involve a zoom into or out of a selected portion of a 
web page, they can be made to work quickly on such thin client computers by use of a layout 
caching scheme of the type just described. 

FIG. 1 18 shows the view of a standard web page that has been laid out at a virtual 
resolution of 640 by 480 and then downscaled and subpixel-optimized for display on a 320 by 240 
screen. Such content is readable by those with good eyes at the distance at which most people 
commonly use handheld computers. However the content of most Web pages can be made even 
easier to read if it is displayed at a larger size. Since most Web content is laid out in a plurality of 
columns, it would often be desirable to be able to quickly zoom a display to the top of a portion of a 
column at which a user would like to start reading. In the interface shown in FIG. 1 19 and 120 a 
user can do this by dragging the pointing device 1 1902 across the desired text column at a vertical 
position that the user would like displayed near the top of the display screen in the zoomed view. 
When the display is in the mode to perform this type of zoom, a horizontal linear drag of the type 
shown in FIG. 1 19 will cause the display to scale the width of the web page layout indicated by the 
drag to fit the width of the screen. In the example shown in FIG. 119 this user input would cause 
the display to be zoomed as shown in FIG. 120. 

Preferably the user interface also allows a user to drag a selection box around an area in 
the web page layout shown on the screen, and the system will zoom the display of the web page 
so that the selected area in the web page fits the screen. 

It is also preferred that in such drags the user be allowed to drag the pointing device 
across a boundary associated with a given edge of the screen, and if this is done the portion of the 
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web page shown on the screen will scroll in response, to allow the user to select to perform a zoom 
to fit to a width, height, or area within the web page layout that is too large or improperly positioned 
to fit totally within the screen as the start of such a drag. If such a drag selects a portion of the 
layout too large to fit on the screen at the scale factor displayed during the drag, it would change 
the scale factor so as to decrease the size at which text and images were displayed. 

When we say that a selected width, height, or area in the layout is scaled to fit the screen, 
we mean that it is scaled to have its largest dimension range between two thirds and the full 
corresponding dimensions of the screen. Normally it would be preferred that such scaling make 
the selected length or area have a largest dimension that ranges from eighty or ninety percent to 
the full corresponding dimension of the screen. 

FIGS. 121 to 128 illustrate a feature of the invention called zoomclick. This feature allows 
a user to more easily and accurately select items within a screen that is seen with a low resolution, 
that is very small, or that is being used with a pointing device that can not easily positioned with 
accuracy relative to desired screen locations. This is particularly useful when dealing with cell- 
phone sized screens, with touch screen devices using fingers as a pointing device, and/or with 
touch screen devices used in an environment such as a moving car in which it is difficult to 
accurately place the pointing device. 

In zoomclick when the user clicks down at a given location in a screen, the portion of the 
screen upon which he or she has clicked is shown at an expanded scale. The user is then free to 
navigate in this expanded representation with the pointing device held down until the pointing 
device is in the desired location. At this point the user can stop pressing down, and release the 
pointing device, causing the current location at the time of the release to be treated as the selected 
location for purposes corresponding to a traditional graphical user interface click. 

With zoomclick double clicks can be represented in different ways. One of the easiest is 
merely to record a quick secondary click and release shortly following a zoomclick and near the 
same location as the zoomclick as converting the zoomclick into a double-click. 
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Although other embodiments of zoomclick can operate differently, the preferred 
embodiment causes pointer movement in the enlarged view during the down click in a zoomclick to 
occur at the same rate as normal pointer navigation in such a scaled view. This means, for 
example, that in zoomclick that doubles the display resolution a user will have twice the pointing 
resolution as he otherwise would. It is also preferred that when a user gets to or near the edge of 
the screen while moving the pointer during a sustained down click in zoomclick mode the image 
will scroll to enable the user to navigate the entire page in this mode. 

In the example of FIGS. 121 through 128, a clamshell cellphones/computer 120C is 
shown. In this example, it is assumed that the cellphones has a 320 by 240 whole pixel resolution 
and color subpixel addressability. Of course in other embodiments of the invention other 
resolutions could be used. For ease of use, the cellphone is assumed to have a touch sensitive 
screen that can be operated by users finger. In other embodiments other types of pointing devices 
can be used with zoomclick, including touch pads, eraser-tip pointers, mice, navigation buttons, 
and ultra-sonic and visual touch screen pointer position detectors. 

FIG. 121 shows the cellphones with the same image of the priceline.com Web page shown 
in FIGS. 11 and 110. 

FIG. 122 shows what happens when, while in zoomclick mode, the user tries to press his 
or her finger 12102 down upon the screen to select the same text entry field 1 1000 discussed . 
above with regard FIG. 110. It is often difficult to estimate in advance the exact location at which a 
cursor will be placed when one touches one's finger to a touchscreen. Zoomclick helps with this, 
since it allows the user to see the position of the cursor 1 70 that results from a finger touching the 
touch screen before any selection is made. It also shows the screen at a larger view scale to make 
it easier for the user to position the cursor 12204 on the desired link or control, such as the desired 
text entry field 1 1000 shown in FIG. 123. 

Once the user removes his finger from the touchscreen with the cursor in the position 
shown in FIG. 1 23, the pop-up keyboard 1 1 1 02 appears, as shown in FIG. 1 24, just as it did in 
FIG. 111, described above. 
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As shown in FIG. 125, when the user, while in zoomclick mode, tries to touch a desired 
letter in the pop-up keyboard 1 1 102, in this case the letter "b," the image of the portion of the 
keyboard touched increases its scale. In the example shown in FIG. 125 the user has not exactly 
position the cursor 12204 at the desired location at the time of his initial pressing of the 
touchscreen. The user can easily correct this problem in zoomclick mode by dragging his finger 
12202 to position the cursor 11102 at the location shown in FIG. 126. If the user then removes his 
finger from the screen, the screens image will revert to its normal scale, which enables the entire 
pop-up keyboard 1 1 102 to be shown to allow rapid selection of the next character, no matter where 
it might lie within the keyboard. 

As shown in FIG. 127, the letter "b" selected in FIG. 126 is shown as having been entered 
in the pop-up keyboard's text entry field 1 1 1 04. 

If the user continues selecting characters by the process shown in FIGS. 125 through 127 
he or she will be able to enter an entire string of text followed by the selection of the enter key, 
which will cause the desired text to be entered at the desired location in the web page, as indicated 
by FIG. 128. 

In many embodiments of the invention, the zooms used in zoomclick involve expanding the 
bitmap previously shown on all or part of the screen by an integral ratio, such as 2x or 3x. This 
allows such zooming to be performed virtually instantaneously, even by relatively low powered 
processors, making zoomclick a very rapid user interface. 

FIGS. 129 through 137 illustrate an aspect of the invention that enables a user of a Web 
browser to select a portion of text from a web page to be the re-flowed, or re-laid out, across line 
boundaries at a substantially larger scale factor. Such re-flowing of text is particularly useful on 
displays having small screens, since it allows selected Web text to be displayed with much larger 
fonts, while at the same time allowing entire lines of such text to fit within such screens. This 
enables such lines to be read quickly, without the need to repeatedly horizontally scroll back and 
forth to read successive lines of such text. 
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Regardless of how high the resolution of a small screen display is, the human eye can only 
see what it displays at a relatively large resolution unless the display is held relatively close. This 
aspect of the invention enables Web text to be display wrapped across lines that fit within the width 
of a display at a relatively large scale factor. For example, it allows the user of a handheld 
computer with a four inch diagonal screen to display text at a sufficiently large scale to be seen by 
a group of people who are standing five or six feet away. Similarly it would allow a user to view 
text on cellphone or wristwatch sized display without having to hold them close to his or her face. It 
can also be used with normal sized computer display screens to display Web text to people who 
are at a relatively large distance from the screen, or who are visually impaired. 

FIG. 129 provides a highly simplified pseudo code description of programming 12900 that 
could be used by a client computer to redisplay Web text according to this aspect of the invention. 

It should be understood that this aspect of the invention is not limited to use on client 
computers. In fact, with modification this aspect of the invention can be used in viewing visual 
output generated by applications other than Web browsers, such as in systems of the type 
described below with regard FIG. 140 and 141. 

Many web pages are designed to have their text laid out in different columns, that is, in different 
horizontal positions relative to such a layout. A web page can indicate such different desired 
horizontal displacements in multiple different ways, including the use of tables and frames. In a 
preferred embodiment such text re-flow is used with a web browser that has the option of 
displaying a web page's text in such a multi-column layout to reflects such indications of different 
desired horizontal displacements. If the user selects an area of such a multi-column web page 
layout for text reflow in a single column at a new scale factor, function 12902 causes function 
12904 through 12908 to be performed. 

In some embodiments of the invention such a selection is made by dragging a pointing 
device, such as the stylus 1 1902 shown in FIG. 130 across the portion of the width of a web page 
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that the user desires to have re-flowed at a larger scale. This is similar to what was discussed 
above with regard to FIG. 119, except that the method currently being discussed allows text to be 
re-flowed across line boundaries, letting a selected column of text be displayed with much larger 
fonts while at the same time allowing whole lines of such text fit within the screen. 

Function 12904 of FIG. 129 selects all strings and corresponding underlining (i.e., labeling 
of text as being a link) in the layout of the current web page that are substantially within the 
selected layout area. 

FIG. 131 illustrates the top portion of the layout 10206A, similar to that shown in the 
bottom half of FIG. 117, of the web page shown in FIG. 130. In FIG. 131 the dashed rectangle 
13102 represent the portion of .the web page's layout corresponding to the column selected by the 
user in FIG. 130. 

In some embodiments of the invention, a string will be considered to be within the selected 
area if a substantial portion, such as two-thirds or three-quarters of its length fits within the area 
selected by the user. For example, in FIG. 130, the user intended to select the text at the right 
hand portion of the screen displayed in that figure. However, in the example of FIG. 130 the user 
failed to exactly select the width of that intended column with the drag of the stylus. Nevertheless, 
because function 12904 selects all strings that are substantially within the selected area, the text 
re-flow will operate as if the user had selected exactly the intended column. 

FIG. 132 illustrates an initial portion of the strings in the layout of the web page shown in 
FIG. 131 that fall within the selected area. In this figure, underlining indicates portions of text that 
correspond to links. 

Once all of the strings in the selected area have been selected, function 12906 labels any 
groups of one or more successive strings whose closeness in the layout or other characteristics 
indicate they are part of the same paragraph. This is indicated in FIG. 132 by the paragraph 
brackets 13202. 
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As indicated in FIG. 132, this method might not detect all groupings of characters that are 
paragraphs, but it does detect many of them without the need to reference the HTML 
corresponding to the text. In the embodiment of the invention being described, such HTML is 
stored on the proxy server, meaning that such an access would require the delay associated with 
communication between the client computer and a proxy server. In other embodiments, 
particularly those in which the client computer has a full browser resident upon it, or those having a 
high access bandwidth link to a proxy server, access to the HTML code could be used to more 
accurately determine how the selected strings should be grouped in paragraphs. In other 
embodiments, the layout information downloaded to a proxy server could contain any information 
about paragraph boundaries contained within a web page's HTML. In some embodiments, such 
re-flowing of text could be performed on a proxy server, rather than on a thin client displaying such 
re-flowed text, enabling the proxy server to directly access any information about paragraph 
groupings contained in a web page's representation. 

Once the selected strings have been grouped into paragraphs, function 12908 re-flows the 
text of each paragraph using the selected expanded scale factor across the width of the display 
screen (or display window if the image is being shown on less than an entire display screen). 

This text re-flow process is illustrated in FIG. 133, in which the strings in the top portion of 
FIG. 132 are laid out at a new scale factor. In the example being shown, the text of FIG. 130 has 
been selected to be re-flowed at twice its original size within the same screen. Preferably the user 
interface of the thin client allows the user to select a plurality of different scale factors for use with 
the selected-text-reflow function, ranging from rather modest to rather extreme increases in font 
size. 

In FIG. 133 underlining is used to represent something different than it does in FIG. 132. 
In FIG. 133 the text on each line that came from a common layout string in FIG. 132 is shown with 
continuous underlining. Underlining gaps between portions of text on the same line in FIG. 133 
that come from different strings in FIGS. 132 are exaggerated to make their difference more readily 
visible. In FIG. 133 all of the individual strings from the original layout shown in FIG. 132 that have 
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been wrapped across a line boundary are indicated by a arrow from their portion on one line to 
their following portion on the next line. 

FIG. 134 provides a schematic illustration of how the selected strings of the original layout 
shown in FIGS. 130 and 131 look once they have been re-flowed at approximate twice the size on 
the thin client's screen. As can be seen by looking at FIGS. 134, such a text reflow makes Web 
text much easier to view from a distance. Reflowing the same text at 4x or 6x instead of 2x would 
make it possible to show the same web content to people at quite a distance from the display 
screen. 

FIGS. 135 through 137 illustrate another method that can allow a user to select a portion of 
text to be re-flowed. 

FIG. 135 illustrates a portion of a web page having a central column of text into which one 
or more portions of other text intrude. 

FIG. 136 illustrates how the user obtained a zoomed out view of the entire web page's 
layout. In many actual embodiments, text greeking would be used to indicate portions of text too 
small to be represented in such zoom-out views as individual characters. Such a zoomed-out view 
could be generated quickly on thin client computers such as those described above with regard 
FIGS. 115 through 1 17 in which a web page's entire layout was stored on the thin client, itself. 

In FIG. 136 the user has selected a mode that allows her or him to define a polygon 
shaped area upon the zoomed-out web page view by clicking the display of the web page at 
corners in such an area. Once this is been done, the selected area will be used by the function 
12904 shown in FIG. 129 to select which text is to be re-flowed. 

FIG. 137 illustrates how the selected text will appear once it has been reflow and 
displayed. 
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FIGS. 138 and 139 provides more description of the font server 230 described above with 
regard FIG. 2. 

FIG. 138 corresponds to FIG. 2, except that in it there are a plurality of the client browsers 
200 each of which accesses content from one or more servers 220 through the same proxy server 
210 and each of which accesses fonts from the same font server 230. 

This is because the software sold, licensed, or distributed for use in each of the thin client 
browsers has been programmed to seek fonts such clients do not have from the same font server 
230 and to make Web requests through the same proxy server 210. Of course, in other 
embodiments of this aspect of the invention the thin clients could be programmed to select which of 
a common plurality of proxy servers to use based on such factors as their geographic location, or 
their Internet service provider. Similar considerations could be used by the thin clients to select 
from which of a common plurality of font servers they are to request and receive fonts. 

FIG. 139 provides a highly simplified pseudo code description of programming 13900 that 
can be used on a font server of the type illustrated in FIG. 138. This font server could also be used 
to provide fonts for normal browser computers, as well as for computers running applications other 
than Web browsers. 

If the font server receives an HTTP request from a computer for a character of a particular 
font, function 13902 causes steps 13904 through 13922 to be performed. 

The particular embodiment of font server code shown in FIG. 139 is designed for use with 
a protocol that specify each character desired for a particular font at a particular size with a 
separate HTTP request. It specifies the desired font, font size, and character as part of a URL 
pathname. Of course, in other embodiments font servers could allow HTTP request to specify 
more than one character, more than one font, and/or could specify fonts other than with URL 
pathnames. 
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In systems that request each character-font shape separately, it is preferable that the 
HTTP protocol 1.1 or later be used, since it allows multiple HTTP request to be handled by a server 
from a given client computer without having to open and close a separate connection for the 
handling of each such request. 

In the embodiment of the invention shown in FIG. 139, if the font server determines that it 
currently has stored a font bitmap corresponding to the URL pathname specified in the request, 
function 1 3904 causes function 1 3906 to send that font in an HTTP response to the network 
address from which the URL request came, and then function 13908 charges an account 
associated with a transaction. Such a downloaded font could be either a font bitmap or a font 
outline description. 

Such a charging of an account is not used in all embodiments of the invention. In some of 
those in which it is used, the account charged is one associated with the computer to which the 
font is sent. In other embodiments, the charge is to an account of a party associated with the web 
page that included a specification for such fonts. In yet other embodiments, the charge is to an 
account associated with a proxy server of the type described above, or to a user of the services of 
such a proxy server. 

If the requested font is not in the font server's storage and it is a bitmap for which the font 
server has a corresponding outline font, function 13910 causes function 13912 through 13922 to 
be performed. 

Function 13912 generates a font bitmap having the attributes, such as size and possible 
transformation, indicated by the font pathname of the HTTP request. This function includes 
determining if the requested font's pathname indicated that a subpixel-optimized version of the font 
is desired. If so, function 13914 and 13916 generate a subpixel-optimized version of the font, 
preferably using the non-linear color balancing method described above with regard FIGS. 55 
through 96. 
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Once the font bitmap has been created, function 13918 sends the bitmap over the network 
in an HTTP response to the requesting address. Function 13920 caches the font bitmap at an 
address corresponding to the pathname specified in the request. Function 13922 charges an 
account associated with the transaction, as discussed above with regard to function 13910, in 
embodiments where such charging is performed. 

FIG. 140 illustrate that certain aspects of present invention can be used to enable a thin 
client computer 200 to display digital content corresponding to the text and the images generated 
as screen output by one or more applications running on upon a remote computer 14000. Such 
applications can include Web browsers, spreadsheets, word processors, database programs, or 
virtually any other type of software capable of generating screen displays. 

The remote computer includes remote screen generator programming 14006, which 
includes hooks in the dispatch table 14008 of the remote computer's operating system 14004. 
These hooks intercept calls made by one or more of the applications 14002 to the operating 
system to draw text, shapes, lines, control objects, and bitmap's to a screen at a given display 
resolution. In some embodiments, such draw commands will actually cause content to be 
displayed on a screen associated with a remote computer, in others there will be no screen at the 
remote computer, and thus such draw commands will be made to a virtual screen. In the text that 
follows, for purposes of simplicity, I will refer to the video space to which these application thinks 
they are displaying graphic output and receiving user input on a given client computer as a virtual 
screen. 

When one of the applications 14002 request the operating system to draw a display 
element, that call is intercepted by one of the hooks in the operating system's dispatch table, so as 
to make a corresponding call to a corresponding routine 14010 of the remote screen generator. In 
a manner similar to that described above with regard to FIGS. 102 and 106A through 106C, this 
causes a download display list 1021 2A to be created that is substantially similar to the display list 
10212 described above with regard to FIG. 102 and the figures that follow it. In many 
embodiments of such a system calls to bitmap draw routines will cause scaled-down subpixel- 
optimized images of the type described above to be generated for download, and calls to measure 
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string and string draw routines will involve font substitution and the font metrics of subpixel- 
optimzed or other anti-aliased fonts, including fonts at sizes as small as 8 pixels per em or less. 

A zoom, scroll, and virtual layout control 1412, corresponding to the controls 10214 
through 10218 shown in FIG. 102, controls the mapping of the thin client's view window into the 
virtual screen and, thus, the display scale factor at which the elements drawn by an application into 
the virtual screen are drawn and positioned in the download display list 10212A. Preferably this 
includes subpixel optimization of image bitmaps, and font substitutions of the type described above 
with regard to FIGS. 106A through 106C. 

Once the download display list has been created for a given virtual screen, it is 
compressed and downloaded to the corresponding client computer, which then draws it upon its 
screen in much the manner described above with regard to FIGS. 109A through 109C. 

In some embodiments of the invention, individual draws to the virtual screen will have 
corresponding draw commands downloaded to the thin client. This, can be used to speed the rate 
at which minor changes to the thin clients screen can be made in response to corresponding 
changes to the virtual screen. 

In the embodiment shown in FIG. 140, user input associated with screen locations are 
uploaded to the remote computer from the thin client, and they have their screen coordinates 
transformed to reflect the mapping between the thin clients view window and virtual screen. Once 
this is done such events are placed in the event queue 14014 of the remote computers operating 
system with their transformed screen coordinates so the associated application 14002 will respond 
to that event as if it had been entered upon the remote computer's corresponding virtual screen. 

Many of the techniques used by screen sharing applications, such as LapLink or 
pcAnywhere, sold by LapLink, Inc., 18912 North Creek Parkway, Suite 100, Bothell, WA, USA 
9801 1 , or pcAnywhere, Symantec Corporation, 20330 Stevens Creek Blvd., Cupertino, CA 95014, 
respectively, can be used in conjunction with an embodiment of the invention of the type shown in 
FIG. 140. In fact, when the remote computer in that figure has its own screen, the embodiment 

BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 617-494-1722 or 617 497-6222 Page 110 



shown in FIG. 140 can be used to perform screen sharing between the client computer and the 
remote computer. 

It should be appreciated that in embodiments in which the client computer has a 
reasonable amount of a computational power, the client and the remote computers can operate in 
a peer-to-peer manner. The remote computer can be a dedicated application server computer or it 
can be any other type of computer, such as a personal computer, including a desktop computers, 
laptop computers, or tablet computers. 

FIG. 141 illustrate an embodiment of the invention that is somewhat similar to that shown 
in FIG. 140, in that it uses hooks into the dispatch table 14008 of a computer's operating system 
14004 to intercept operating system calls made by one or more applications 14002 in order to 
cause the screen displays generated by such applications to be scaled-down and/or subpixel- 
optimized according to aspects of the invention described above. It is different from the client- 
server embodiment shown in FIG. 140, in that it is designed to run on one computer system 14100, 
shown in FIG. 141. The programming of this system can be used on a computer having a low 
resolution screen but enough power to run desktop applications, to run such programs without 
modification as if their output were being displayed on a virtual screen having a higher resolution. 
It can also be used to draw a scaled-down version of an application's output screen on a portion of 
a larger screen. 

In this embodiment, the applications 14002 are told they are running on a screen with a 
virtual resolution larger than the resolution at which their output is to be displayed on the screen 
10220A. When such an application makes a call to the operating system to draw an element to a 
screen, the hooks placed in the OS dispatch table 14008 cause a corresponding draw routine 
within a part 1401 OA of a scaled subpixel-optimized screen generator program 14006A to be 
evoked. This substitute draw routine draws a corresponding element to a virtual screen display list 
10206B, including creating scaled-down and/or subpixel optimized bitmaps as has been described 
above. It also causes any portions of such screen elements drawn into the part of the virtual 
screen that fits within the current view window 10210C to be immediately displayed in a scaled- 
down and/or subpixel-optimized manner on the display screen 10220A of the computer 14100 by 
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means of calling draw commands in the operating system, or by directly drawing to that screen 
themselves. 

When an application program calls the operating system for a measure string commands, 
that commands is likewise intercepted so the call returns font metrics for a substituted font size in 
the manner described above with regard to functions 10608 through 10618 of FIG. 106A. 

A screen event input into the computer's screen is taken from the operating system's event 
queue and passed to an event position scaler, which transforms the screen coordinates at which 
such an event was generated on the screen into a corresponding position in the virtual screen's 
layout represented by the display list, using the mapping of the view window into that virtual screen 
to control such a transformation. Once the coordinates of the event have been appropriately 
transformed, the event is returned to the operating system event queue so the operating system 
will cause the application to respond to the event as if it have been entered onto the virtual screen. 

An embodiment of the invention of the type shown in FIG. 141 would allow a user of a 
computer to subpixel-optimized, scaled-down, zoom, and perform selected text reflow upon 
screens generated by standard computer applications 14002, even if they have not been designed 
to support such functions. 

In other embodiments of the invention not shown, the operating system of a computer can 
be modified to include functionality of the type shown in the scaled-subpixel-optimized screen 
generator 14006A shown in FIG. 141 . In yet other embodiments of the invention, application 
programs 14002, including browser programs, can be modified to support all or a part of such 
functionality directly. 

FIG. 142 illustrates how the embodiment of the invention shown in FIGS. 102 and 140 can 
be used to allow thin client computers, such as the thin client computers 200A through 200D 
shown in that figure, to be used to access Internet content or application programs over wireless 
network. 
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In this figure the computers 200A through 200D correspond to the thin client computer 200 
shown in FIGS. 102 and 140. The computer 200A is a handheld computer. The thin client 
computer 200B is a cellphone. The thin client computer 200C is a wristwatch computer. The thin 
client computer 200D is a headmounted computer, or headmounted display for a portable 
computer. Each of these client computers can have a subpixel addressable display. 

At the time of the filing of this application it is currently possible to manufacture screens for 
each of these types of devices having resolutions high enough for use by most aspects of the 
present inventions. For example, at the current time it is possible to manufacture a 320 x 240 color 
LCD display with a diagonal measurement of 2 inches or less. Organic LED devices can currently 
be manufactured with even higher resolutions. In the near future, the cost of such small screens 
should come down, and their availability and resolution should go up. 

All of the thin client computers shown in FIG. 142 have wireless transceivers that enable 
them to transmit and received information of the type described above with a remote proxy server 
computer 210 of the type shown n FIG. 102 or a remote application server 14000 of the type 
shown above with regard FIG. 140. Such transceivers can be wireless LAN transceivers for 
communicating with a wireless LAN transceiver 14204 or digital cellular wireless transceivers for 
communicating with a wireless Internet transceiver 14202, or preferably a transceiver that has 
been designed to communicate with both types of wireless transceivers. In other embodiments 
other types of wireless communication, such as Bluetooth or infrared communication, can be used. 

The remote computers 14000AA through 14000AC shown in FIG. 142 correspond to the 
remote server computer 14000 shown in FIG. 140. 

The remote application server computers 14000AA shown in FIG. 142 represent laptop, 
desktop, server or other types of computers that can be programmed to operate as a remote 
application server computer 14000. The subpixel-optimized application server 14000AB is a 
remote computer of the general type illustrated in FIG. 140 that is designed to run applications for a 
plurality of thin client computers connected to a LAN or WAN associated with such clients. The 
remote computers 14000AA and 14000AB can communicate with thin clients over a private local 
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area wireless transmitter 14204, or can communicate with them over the wireless Internet as 
indicated by the numerals 10222 and 14202. 

The subpixel-optimized application server 14000AC is an application server similar to 
server 14000AB, except that it is connected directly to the internet to allow multiple thin client 
computers 200 to use applications over the Internet by means of the wireless transmission network 
indicated by the numeral 14202. 

In FIG. 142 a proxy server 210, of the type described above with regard to FIG. 102, is 
shown connected to the LAN or WAN 14204. This, for example might be a proxy server intended 
to handle Web browsing that the Corporation wishes to keep off the Internet. It should be 
understood that other such proxy servers, such as those operated by companies providing 
commercial proxy serving services, would normally be connected directly to the Internet 10222 
shown in FIG. 142 as well. 

The system illustrated in regard FIG. 142 allows small computers that can be conveniently 
carried at virtually all times to access and display web pages and the output of most application 
programs. At the time of filing this application, the bandwidth of relatively inexpensive wireless 
LAN transceivers, such as the LAN transceiver 14204 shown in FIG. 142, is fast enough to allow 
thin clients of the type shown in FIG. 142 to view web content or the output of application programs 
almost as rapidly as one could view such digital content on a desktop computer connected to a 
cable modem. And this is on a machine that can be carried one's pocket, or on one's wrist, or as 
part of one's glasses, and which can be capable of accessing such media within several seconds 
after being turned on. 

At the digital cellular bandwidth commonly available in America at the time of filing this 
application, it will normally take several seconds to download the entire text of the most web pages, 
and longer to download the web page's images. Of course many embodiments of the present 
invention start to display text as soon as part of it is received, allowing the user to start seeing part 
of a downloaded page very quickly. 
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As of this filing new, higher speed, digital cellular systems have been developed that are 
capable of providing bandwidths in the range of hundreds of thousands or millions of bits per 
second. Once such higher speed systems become commonly deployed, users of the invention will 
be able to read and interact with web pages and application screens on small, portable devices, 
that can be used within seconds of being turned on most places they travel, with almost as much 
speed and convenience as if accessing them on a desktop or laptop through a DSL or cable 
modem connection. 

FIGS. 143 and 144 provide two views of a handheld computer 200A capable of functioning 
as a thin client for either proxy servers of the type described above with regard to FIG. 102 or a 
remote application server computer of the type described with regard to FIG. 140. 

In FIG. 143, the computer is shown in the portrait orientation in which it has been designed 
for use. The native operating system on the computer is designed to draw fonts and graphical user 
interface elements in this portrait orientation. This is the manner in which many of the handheld 
computers sold at the time of the filing of this application have been designed and built. For 
example, there are multiple such handheld computers on the market today that have subpixel 
addressable screens with a 240 x 320 whole pixel resolution. Many of these computers also have 
subpixel striping that runs in a horizontal direction when the displays are in their intended portrait 
orientation. 

Unfortunately, such a portrait orientation does not provide the type of landscape aspect 
ratio with which most people are used to using computers, and for which most web pages have 
been designed. Furthermore, in the case where such computers have horizontal subpixel striping, 
such striping provide all of its potential increase in subpixel resolution in the vertical direction. 
Unfortunately, the display of text tends to benefit substantially more from an increase in horizontal 
resolution than it does from such an increase in vertical resolution. 

For all these reasons, many embodiments of the invention that use such portrait-oriented 
machines are designed to use them when they have been rotated by 90 degrees, as shown in FIG. 
•144, so they will have a landscape aspect ratio more like that of the layout of most computer 
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screens, and so that their subpixels will provide an increase in horizontal resolution that is most 
useful for displaying text. In many such embodiments the thin clients 200 described above are 
portrait-oriented PDA ("Personal Digital Assistant") computers that have been rotated to be viewed 
as landscape oriented machines and in which scaled-down font and image bitmaps are drawn in a 
landscape orientation. In such embodiments that use subpixel-optimization, the web page is 
displayed using image and/or font bitmaps that have been subpixel optimized for the spatial 
arrangement of subpixels within each pixel that appear when such pixels are viewed in the 
landscape orientation, rather than the orthogonal arrangement of subpixels that appears when the 
pixels are viewed in the portrait orientation in which such PDA's were designed for use. 

The FIG. 145 is a highly simplified pseudocode representation of how some aspects of the 
present embodiments can be used to respond to requests to draw basic shapes - such as 
rectangles, ovals, lines, and curves -- using subpixel optimization. Such functionality can be used 
in applications of many different types, in operating systems, and in thin client software. 

In the example of FIG. 145, the pseudocode shown relates to a rectangle draw function 
14500, which could, among other uses, be used in place of the rectangle command 10918 
described above with regard FIG. 109A. Such a routine is evoked by a call to draw a rectangle that 
has its position, width, and/or height defined at higher resolution than the whole pixel resolution of 
a subpixel addressable screen on which is to be shown. In response, the function 14502 uses a 
subpixel-optimization routine to render the image of the rectangle defined at such a higher 
resolution, at subpixel resolution. This can be done using virtually any subpixel optimization 
scheme, but for monochrome rectangles a bicolor optimization scheme with non-linear color 
filtering, such as that described above, will often tend to provide the highest perceived spatial 
resolution. 

FIG. 146 is a highly simplified pseudocode representation 14600 of code 14602 that 
operates on a server and/or proxy computer and code 14604 that can be run on a client computer, 
including a thin client computer, to allow applets downloaded from the server to draw subpixel- 
optimized screen elements on the screen of the client. 
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In such an embodiment, a function 14606 of the client requests media from the server. 
The server responds in function 14608 by downloading media, or data, including one or more 
applet programs that can run on the client computer. In function 14610 the client computer 
receives the media including the applets, and function 14612 loads and runs the applets. In 
function 14614 the applets draw subpixel-optimized elements to the subpixel addressable screen 
on the client computer. 

The applets can draw subpixel-optimized elements either by copying or generating 
subpixel-optimized bitmaps, by rendering text with subpixel-optimized fonts, or by drawing subpixel 
optimize shapes, such as the shapes of vector defined graphics or relatively simple geometric 
shapes, such as lines, rectangles, and ovals. 

FIGS. 147 and 148 illustrate how subpixel optimization can be applied to rollover images 
and GIFF animations, respectively. 

In the subpixel optimization routine 14700 shown in FIG. 147, both a non-rollover image 
14702, which is to be displayed when a pointing device is not detectably over the portion of the 
screen associated with the images, and a rollover image 14704, which is displayed when the 
pointing device is detectably over that screen portion, are both downscaled and subpixel-optimized 
by a function 14706. This produces a scaled subpixel-optimized non-rollover image 14708 and a 
scaled subpixel-optimized rollover image 14710. Then a function 14712 is used to select which of 
these two subpixel-optimized images is displayed based on whether the pointer is detectably over 
their associated screen area or not. This makes the two subpixel-optimized images act as a 
combined "rollover" graphic. 

In other embodiments of this aspect of the invention, a similar technique could be applied 
to two images that are associated with a button, one displayed when the button is not being 
pressed, and another displayed when the button is pressed. 

The method 14800 shown in FIG. 148 is similar to that described above with regard FIG. 
147. It takes each separate image 14802 through 148906 of a GIFF animations and subpixel- 
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optimizes it in a function 14808 to produce a corresponding set of scaled-down, subpixel-optimized 
GIFF animations images. Then function 14816 displays the subpixel-optimized images in 
substantially the same manner that non-subpixel-optimized GIFF animations are displayed. 

The subpixel optimizations described with regard to FIGS. 147 and 148 can be used with 
other aspects of the invention described above, including in the accessing of web pages on a 
subpixel addressable screen, including those on thin client computers. 

FIG. 149 illustrates a method 14900 for subpixel optimizing 3-D animation. This method 
includes performing a set of functions 14904 through 14908 for each successive frame of the 
animation. 

Function 14904 runs a 3-D animation engine to create a bitmap of the current frame, or at 
least of those portions of the image that have changed since the last frame. This function 
generates such bitmaps at a resolution higher than the whole-pixel resolution at which the 
subpixel-optimized version of such bitmaps are to be displayed. 

Function 14906 then uses techniques, such as those described above, for scaling down 
and subpixel optimizing the frame bitmap, or at least changes made in the frame bitmap since the 
last frame. 

Next function 14908 displays the scaled-down, subpixel-optimized image of the frame 
bitmap, or at least of the changed portion of the frame, on a subpixel addressable screen. 

The method shown in FIG. 149 can be particular useful to allow people to play games, and 
see the images produced by such games at the higher resolution made possible by subpixel 
optimization. It can be used for such purpose on small screen, handheld devices. It can be used 
both with client computers displaying animated images generated on a remote computer, as well 
as with computers that are generating such animated images locally. 
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FIGS. 150 and 151 illustrate one way in which the method of FIG. 149 can be used in a 
client server gaming application. 

FIG. 150 illustrates programming 15000 on a game server computer used in such an 
embodiment. As indicated by the numeral 15002 and 15004, if the game server receives user 
input from one or more game client computers it sends input to the game engine. If such input is 
screen input, it is scaled appropriately to compensate for the difference between the user's screen 
resolution and the space that the game engine associates with screen inputs. 

In function 15006 the game engine computer computes a display list for the current frame, 
or for any changes associated with the current frame to a prior display list. Then function 15008 
has a 3-D rendering program render a frame bitmap corresponding to the display list generated for 
the current frame, or render the changes required to the bitmap of the current frame. Such bitmaps 
are generated at a higher resolution than that of the subpixel-optimized images that are to be 
created by the function 1 501 0. 

If the client is generating different screen images for different clients, the function 15008 
would be performed separately for each of those separate views. 

Next function 15010 scales down and subpixel optimizes the current frame bitmap or the 
bitmaps of current changes to the frame. When the function is scaling down only bitmaps of such 
changes it also correspondingly scales down the screen positions associated with those changes. 

Next function 15012 compresses the subpixel-optimized bitmaps, and if appropriate, their 
locations, and function 15014 downloads the compressed, scaled, subpixel-optimized images and 
any such locations to the client for display. 

FIG. 151 illustrates programming 15100 on a game client designed for use with a 
programming of FIG. 150. 
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Function 15101 receives downloaded images, then function 15102 decompresses them. 
Next function 15104 displays the scaled, subpixel-optimized animation frame bitmaps, or it displays 
bitmaps of changes over the image of the prior animation screen at the locations indicated for 
those changes. This is done on a subpixel addressable display. 

As indicated by numeral 15106 and 15108, when the client receives user input, it uploads 
that input to the game server with any screen coordinates associated with those inputs being 
appropriately translated. 

In other embodiments of this aspect of the invention the distribution of functionality 
between the game server and the game client could be different. In some embodiments, a proxy 
server generally similar to that described above could be used to perform the subpixel optimization 
for display on a thin client of game content originally generated on a game server that is different 
than the proxy server. In yet other embodiments the game client could itself perform the subpixel 
optimization. 

FIG. 152 is a highly simplified pseudo code description of an aspect of the invention that 
allows images having associated transparency maps to be displayed with subpixel optimization of 
both such images and their associated transparency map 

The programming 15200 shown in FIG. 152 includes a function 15202 that produces a 
scaled subpixel-optimized bitmap of a foreground image, that is an image, the display of which on 
top of a background or other prior bitmap is to be controlled by an associated transparency bitmap. 
The subpixel optimization used can be either a bicolor or a multicolor subpixel optimization, or a 
combination of the two. Any method known for producing subpixel-optimized representations of 
images could be used, including those that have been described above. 

Function 15204 produces a subpixel optimization of the image's associated transparency 
map. Preferably a bicolor subpixel optimization is used, since a high resolution source image of a 
transparency map has transparency values that vary along a straight line in 3-component color 
space, that of an alpha value ranging from 0 to 1 . Such source image alpha values correspond to 
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grayscale colors because, if the area of the transparency map source image corresponding to a 
given pixel in the subpixel-optimized output image of that map is covered by a uniform 
transparency value, all of that output pixel's subpixels will tend to have equal alpha values. 
Preferably the bicolor subpixel optimization of the transparency map is created using the non-linear 
color balancing described above. 

Once such a subpixel optimization of a foreground image and its associated transparency 
map has been created, function 15206 displays this combination on a subpixel-optimized display. 
This process includes performing a loop 15208 for each pixel row of the displayed image, which 
includes a loop 15210 for each subpixel of each such row. The function 15210 causes function 
15212 and 15214 to be performed for each subpixel. The function 15212 sets the current alpha 
value to the alpha value of the corresponding subpixel of the subpixel-optimized transparency map. 
Then function 15214 sets the luminosity of the current subpixel to the current alpha value multiplied 
by the luminosity of the corresponding subpixel of the subpixel-optimized foreground image plus 
the prior luminosity value of the current subpixel in the background bitmap over which the 
transparency image is being drawn multiplied by one minus the current alpha value. 

This means that if the foreground image is drawn over a prior bitmap, the extent to which 
luminosity of each of its separate subpixels is derived from the corresponding subpixel value of the 
foreground image, or of the prior bitmap is determined as a function of the corresponding subpixel 
alpha value of the subpixel-optimized transparency map. 

In some embodiments of the invention images with associated transparency maps will be 
scaled and subpixel-optimized on a server or browser computer, downloaded, and then displayed 
by function 15206 on a client computer. In other embodiments of the invention, such subpixel- 
optimized transparency images will be made available on recorded digital media. In yet other 
embodiments of the invention they will be generated by the same computer that displays them. 

In other embodiments of the invention subpixel-optimized foreground images could be 
displayed using alpha values contained in a non-subpixel-optimized transparency map. 
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In some embodiments of the invention lossy color compression will be used to represent 

* 

groups of colors that are perceptually close with one color. Such compression can be performed 
upon one dimensional transparency values, upon three dimensional transparency (i.e., opacity or 
alpha) values of the type described above with regard to FIGS. 60, 96, and 97, or upon color 
values having a transparency component value as an extra color dimension, as well as upon RGB 
component values. In such compressions, it is generally advisable to prevent transparency values 
or component color values representing an alpha of one or zero, or values very close to one or 
zero, from being represented by transparency values further from one or zero, respectively. This is 
because the eye is more sensitive to slight changes in opacity at the extremes of the transparency 
range than it is to such changes elsewhere in that range. 

Subpixel-optimized images with transparency maps can be used on subpixel optimized 
displays for all the purposes for which non-subpixel-optimized images are used with transparency 
maps. This includes use in animations and in web page layouts. 

FIGS. 153 through 162 are highly simplified pseudo code descriptions of aspects of the 
invention relating to subpixel optimization of video and/or animation. Such subpixel optimization 
can be used in the context of Web browsing as well as in virtually any other context in which video 
and animations are used. 

FIG. 153 represents programming 15300 used to subpixel optimize video represented 
using interpolation between video key frames. This programming includes a function 15302 that is 
used in the case where the video to be subpixel-optimized is received in compressed format. It 
decompresses such video, so it can be subpixel-optimized. 

Function 15304 scales down and subpixel optimizes the keyframes of the video. Function 
15306 scales down but does not subpixel optimize interpolated changes between keyframes. In 
some embodiments of the aspect of invention shown in FIG. 153, such interpolation changes could 
be subpixel-optimized, but there is little benefit from doing so, since such changes appear so 
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rapidly on a screen that their subpixel optimization would not be noticeable, and avoiding their 
subpixel optimization reduces computational overhead. 

Then function 15308 displays the scaled down video on a subpixel addressable display 
with the subpixel-optimized keyframes and the non subpixel optimize interframe interpolation. 

In other embodiments of the invention, this concept of only subpixel optimizing portions of 
video that will be on the screen at one location long enough to be clearly perceived could be used 
in other ways. 

FIG. 154 illustrates programming that can be used to subpixel optimize video represented 
totally or partially by sequences of sub-whole-frame image elements that are to be drawn to a 
display frame. Commonly such video will also include whole frame images, and will use a 
sequence of sub-whole-frame draws to incrementally changes screen as needed to represent 
motion of one or more objects within it. This would include animation of the type described above 
with regard FIG. 149. It can also include various forms of video compression, including video 
having keyframes and interframe interpolation of the general type described above with regard 
FIG. 153. 

The programming of FIG. 154 includes a function 15402 used where the video to be 
subpixel-optimized is received in compressed format, in which case that function decompresses it. 
Next function 15404 scales and subpixel optimizes any frame images contained in the video, 
scaling them down by a display scale factor. Then function 15406 scales and subpixel optimizes 
any change bitmaps, scaling both the size of such images and their location by the scale factor. 

Functions 15407 and 15408 repeatedly display on a subpixel addressable screen any 
scaled subpixel-optimized video frame in the video sequence. After the display of such a video 
frame it displays any of one or more scaled, subpixel-optimized change bitmaps over the bitmap of 
that frame at the scaled position associated with that change bitmap by the function 15406. 
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It can be seen that the method of FIG. 154 enables subpixel-optimized video and 
animation to be drawn in a manner that reduces the amount computation required for subpixel 
optimization, since it does not require the subpixel optimization of an entire frame each time a 
change is made to its video image. 

FIGS. 155 and 156 illustrate two different methods in which subpixel-optimized images that 
move relative to a frame can be displayed. 

FIG. 155 includes programming 15500 that displays an image with fixed subpixilation as it 
moves in whole pixel increments relative to a larger image on a subpixel addressable display. It 
includes a function 15502 that stores a subpixel-optimized image, which can be produced by any 
method, including those described above. It includes a loop 15503 performed for each successive 
frame time. This loop comprises the function 15504 and 15506. The function 15504 calculates a 
movement for the image relative to the larger image. In this movement calculation the position 
calculated for the object at each display frame is rounded to the nearest whole horizontal and 
vertical pixel location and the size and orientation of the image is not altered. The function 15506 
displays the image at the whole pixel resolution location calculated for it by the function 15504. 
Since only one subpixel-optimized bitmap of the image has to be calculated, and that single image 
is repeatedly used as it moves across the screen, this method is quite computationally efficient. 

FIG. 156 describes programming 15600 that displays a moving image with changing 
subpixilation. It includes a function 15602, which stores a high resolution source image of the 
image to be moved. It also includes a loop 15603 performed for each successive frame time. This 
loop includes a function 15604, which calculates the current translation, rotation, and/or 
transformation of the high resolution source image, if any for the current frame. Then the loop's 
function 15606 generates a scaled-down, subpixel-optimized bitmap of the translated, rotated, 
and/or transformed bitmap so produced. This subpixel optimization takes into account the location 
of this transformed bitmap relative to the subpixel array upon which it will be displayed at a 
resolution higher than whole pixel resolution. Then function 15608 of the frame loop displays the 
resulting subpixel-optimized bitmap on a subpixel addressable display. 
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Either of the methods described above with regard FIG. 155 or 156 can be used to display 
sprites in game animation, as well as animated text, or any other type of visual representation that 
is moved relative to a larger frame. 

The method of FIG. 155 tends to provide a less accurate representation of the motion of 
the visual object, but it is more computationally efficient. The method of FIG. 156 provides a more 
accurate visual representation, but is more computationally expensive. 

In some embodiments of the invention a combination of these two methods could be used. 
For example, a small subset of possible mappings between the object and a subpixel array can be 
stored, and as the object moves it is displayed with that one of such stored mappings that most 
closely represents a higher resolution representation of its current location relative to the subpixel 
array upon which it is to be displayed. 

FIGS. 157 and 158 illustrates aspects of the present invention used to optimize the display 
of DVD or an HDTV video by downscaling and subpixel optimizing such video for display on a 
subpixel addressable screen. This is particularly useful when used in conjunction with subpixel 
addressable screens that have a higher subpixel resolution in the horizontal direction than they do 
in the vertical direction, because both DVD and HDTV video commonly have an aspect ratio 
substantially wider than it is high. 

FIG. 159 illustrates aspects of the invention that can be applied to video formats that 
represents subcomponents of video images as separate objects having different attributes. The 
particular example in FIG. 159 involves programming 15900 that subpixel optimizes the display of 
MPEG-4 video. • 

The programming shown in FIG. 159 includes a function 15902 that receives and 
decompresses an MPEG-4 video. It includes functions 15904 and 15906 that use different 
subpixel optimization methods when scaling down different types of objects in the MPEG-4 video. 
This function uses bicolor subpixel optimization, preferably with non-linear color balancing, on 
bicolor objects, and it uses multicolor subpixel optimization on multicolor objects. It's function 
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15908 displays a combination of the bicolor and multicolor objects on a subpixel-optimized screen, 
moving such subpixel-optimized objects relative to the screen as dictated by the MPEG-4 
description, using methods of the type discussed above with regard to FIGS. 155 and/or 156. 

Some aspects of the invention are not limited to such use of different subpixel optimizing 
algorithms for different object types in the MPEG-4 data stream. But the use of such different 
subpixel optimizing algorithms can provide higher perceived resolution for bicolor objects, such as 
text, and thus has the advantage of providing a somewhat betterimage. 

FIGS. 160 and 161 relate to systems in which users access subpixel-optimized video over 
a computer network. 

FIG. 160 illustrates programming 16000 used by a server computer that serves subpixel- 
optimized, scaled-down video. Such a server could be a proxy server that accesses video 
requested by the client from yet another server computer and then downscales and subpixel 
optimizes it before downloading to the client. In other embodiments, the serving of such subpixel- 
optimized video is performed without such an intermediary proxy server. 

The programming of FIG. 160 includes a function 16002 that receives a request for certain 
video from a client computer. In many embodiments, such as the one shown in FIG. 160, the 
request will also describe the horizontal and vertical subpixel resolution for which the video is to be 
subpixel-optimized. In embodiments in which the server is only serving a set of clients having one 
fixed subpixel resolution, such information is not needed as part of the request. 

The function 16004 receives the requested video content. This can be done by accessing 
it from a remote server, as described above; by accessing it from RAM or a mass storage device 
associated with the serving computer; by having such content dynamically generated; or by 
selecting a video fed from some source. 

Function 16006 scales down and subpixel optimizes the received video to the subpixel 
resolution associated with the request of function 1 6002. Then function 1 6008 compresses the 
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subpixel-optimized video and function 16010 download that compressed video to the requesting 
device. 

The compression algorithm used for such subpixel-optimized images can include one 
which has a certain amount of loss without substantially decreasing the increased spatial resolution 
made possible by subpixel optimization, as long as the location of the color values associated with 
any pixel in such subpixel-optimized images are not moved in RGB color space by more than a 
relatively limited color distance. 

FIG. 161 describes a system 16100 that can be used with the aspect of the invention 
described in FIG. 160. This system includes proxy computer code 16100 and thin client computer 
code 161 12, both of which are illustrated by highly simplified pseudocode in FIG. 161 . 

When the thin client receives a user request for certain video, function 161 13 responds by 
sending a request for the video, including the subpixel resolution at which the video is to be 
displayed to the proxy. When the proxy receives the request for such video its function 16100 
causes function 16103 to send a corresponding request for the video to a server from which it can 
be obtained. In many embodiments this will be a server identified in the URL of such a request. 

When requested video is received by the proxy server, function 16104 causes function 
16106 through 161 10 to be performed. Function 16106 scales down and subpixel optimizes the 
video to the subpixel resolution associated with the request from the client; function 16108 
compresses that subpixel-optimized video; and function 16110 downloads it to the client that has 
requested it. 

When the client receives requested video from the proxy, function 161 14 causes function 
161 15 to decompress it, and function 161 16 to display that downscaled, decompressed the video 
on a subpixel addressable display. 

FIGS. 162 through 166 are used to illustrate how aspects of the invention can be used to 
improve the appearance of digital ink. Digital ink is usually a black and white bitmap drawn on a 
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screen in response to a user attempting to write or draw with his or her pointing device. In the 
past, digital ink bitmaps have usually been represented at a whole pixel resolution in which each 
pixel is shown as either black, white, or in some devices a grayscale value. 

One aspect of the invention is the use of subpixel optimization to represent digital ink with 
a higher resolution. When digital ink that is represented within the computer's memory by points 
and lines or curves between such points, the resulting mathematical description of the lines 
between such points can have a much higher resolution than the whole pixel resolution of the 
screen. 

FIG. 162 is a highly simplified pseudocode description of programming that can be used to 
optimize the clarity with which digital ink can be viewed. 

The digital ink code 16200 shown in FIG. 162 includes a function 16202 that responds to 
user input with a pointing device while in digital ink draw mode, by recording the strokes of the 
pointing device as a series of points and curve or lines between such points. Function 16206 
draws ink on the screen using a subpixel optimization of the lines and curves. This can be done 
with virtually any subpixel optimization scheme, but it is preferably done with a bicolor subpixel 
optimization scheme, such, as for example, the bicolor subpixel optimization scheme using non- 
linear color balancing described above. 

FIG. 163 illustrates some digital ink 16302 that has been drawn on the screen of a 
handheld computer 16300. Because this illustration is printed with a printer that can only represent 
whole pixel luminosity values, the digital ink illustrated in FIG. 163 displays subpixel optimization as 
grayscale anti-aliasing. It to be appreciated that when viewed on a subpixel addressable display 
the image would appear even more clear than shown in FIG. 163. 

If the user of the digital ink programming selects to scale up a representation of a portion 
of digital ink, function 16208 causes function 16212 to produce a subpixel-optimized bitmap of the 
digital ink lines and curves, using a bicolor subpixel optimization with non-linear color balancing, at 
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the user selected scaled-up size. Then function 16212 displays that scaled-up image on the users 
screen. 

FIG. 164 illustrates a scaled-up representation 16302A of the portion of digital ink 16302 
shown in FIG. 163. This provides a substantially more clear representation of the digital ink, than 
is produced by merely blowing up the pixilation of the digital ink's representation 16302 shown in 
FIG. 163, as is illustrated by the bitmap 16302B shown in FIG. 165. 

It should be noted that the representation of the bitmap shown in FIG. 165 is actually more 
pleasant to look at than some scaled-up representations of digital ink because the bitmap shown in 
FIG. 163 has been printed with whole pixel grayscale values with anti-aliasing, which is not used in 
some digital ink representations. 

If the user selects to scale down the representation of digital ink, function 16214 causes 
function 16216 to produce a subpixel-optimized bitmap of the ink's lines and curves using bicolor 
subpixel optimization with non-linear color balancing at the selected scaled-down size, and then 
causes function 16218 to display that scaled-down bitmap on the subpixel addressable display. 
The results of such a process is illustrated by the bitmap 16302C shown in FIG. 166. 

These aspects of the invention can be modified to deal with digital ink that has been 
recorded as whole pixels that are either on or off. This can be done by having a routine estimate a 
centerline of each stroke represented by such "on" pixels, and then producing a subpixel optimize 
image of the digital ink's centerline at various scales as described above. A more accurate but 
more computationally expensive approach would be to seek an optimal fit between successive 
portions of such digital ink and a corresponding succession of lines and curves, such as, for 
example Bezier curves. 

In other embodiments, subpixel optimization could be performed on bitmaps that have 
been produced by digital ink drawing by merely performing subpixel-optimized scale ups or scale 
downs upon such bitmaps. 
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Some embodiments of the invention that relate to digital ink could be used with non 
subpixel-optimized displays, by replacing subpixel optimization with grayscale anti-aliasing. 

FIG. 167 illustrates components that can be included in many of the server, client, proxy 
server, thin client, remote, desktop, or other computers referred to above. It should be understood 
that not all of the components shown in FIG. 167 will be in all such computers, and most such 
computers will include other components besides those shown in FIG. 167. 

This figure is provided to make clear that most of the computers used with various aspects 
of the present invention include some type of processor 16716 capable of executing programming 
16702 to cause it to perform the functions of such aspects of the invention and to read and write 
data 16704 according to the methods of such aspects. The present invention relates to not only to 
methods but also to such computer programming and data, as well as to computer systems that 
have been programmed and/or hardwired to perform such methods or to use such data. 

In most such computers the invention's programming will be stored in machine readable 
form in RAM 16706; ROM 16707; or a mass storage device such as a hard drive 16708, floppy 
drive 16709, CD-ROM drive 1671 1 , and/or DVD drive 16713. It can also be stored in machine- 
readable media, such as on a floppy disks 16710, CD ROMs 16712, DVD ROMs 16714, or virtually 
any other type of machine readable storage media. The invention's programming and/or data can 
also be the represented as propagated signals, indicated by the numeral 16719, that can be 
received by the computer through some sort of communication port, such as the network interface 
16720. 

FIG. 168 provides a whole-pixel grayscale representation of a 320 by 240 screen showing 
a small subpixel-optimized font produced using the non-linear color-balance method described 
above with regard to FIGS. 60 through 97. This figure is identical to FIG. 56 except that a portion 
of its text is encircled by dotted lines 16800. 
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FIG. 169 is an eight times blowup of the portion of the bitmap shown in FIG. 168 within the 
dotted lines 16800. It shows that most of the vertical strokes in the font shown in FIG. 168 contain 
color-balance distributions on their left-hand side that blur the clarity of such fonts. This same 
slight blurring occurs in the small fonts shown in FIGS. 1, 10, 11,99, 101,104, 110- 114, 121 - 
128,130, and 144. 

One of the major benefits of the non-linear color-balancing method of producing subpixel 
optimized font bitmaps is its ability to decrease the blurring of character-font shapes by the non- 
linear method that seeks to prevent the distribution of luminosity values where such distribution is 
not needed for color balance. 

Upon observing the undesirable spreading of color values to the left of the main strokes of 
fonts of the type shown in FIG. 169, the inventor of this aspect of the invention sought to see if 
such spreading could be reduced. He tried to determine what the source of such spreading was. 

Referring now to FIG. 170, he found that the algorithm used for creating non-linear color- 
balanced bitmaps was designed to automatically place two padding columns of subpixels 17000 to 
the left of the leftmost subpixel column 17002 in the rasterization of a character-font shape that 
included an actual non-zero coverage value 17004 (i.e., was actually covered by a portion of the 
character-font shape being represented by the rasterization). This was done to provide room for 
the possible spreading of color balancing color values into two subpixel column to the left of the 
leftmost subpixel column containing such a non-zero coverage value. This would be desirable if 
such a leftward spreading was required by the non-linear color balancing algorithm, described 
above, which allows color balancing distribution two pixel to the left of a partially covered subpixel.. 

Unfortunately padding the rasterization subpixel array with only two such subpixel columns 
17000 tends to have the undesirable effect of making the leftmost subpixel column 17002 that 
contains such a coverage value be the rightmost subpixel column of the pixel column containing 
the two padding subpixel columns. In an RGB display this would cause the leftmost subpixel 
column containing an actual coverage value to correspond to a blue subpixel. 
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This is undesirable because it tends to cause pixels in the leftmost pixel column in a font 
bitmap to contain two leftmost subpixels that have no actual coverage value and a rightmost 
subpixel that does include a non-zero coverage value, requiring that non-zero coverage value to be 
distributed to achieve color balancing. This is a reason for much of the leftward blurring of major 
vertical strokes shown in FIG. 169. 

The inventor noted that character-font shapes hinted with systems that allowed boundaries 
of vertical strokes to be positioned in increments finer than the width of a subpixel column had 
often been designed by the individuals who hinted them to start the leftmost edge of their leftmost 
vertical stroke, such as the edge 17100 shown in FIG. 171 only a slight distance into the leftmost 
subpixel column containing non-zero coverage values 17002. This would substantially reduce the 
amount of the non-zero coverage value contained within the subpixel column 17002 that had to be 
distributed by non-linear color balancing, thus greatly reducing undesirable blurring in the subpixel 
optimized representation of the character. 

For example, the inventors found that many of the best hinting combinations, when used 
with such algorithm, cause the first vertical strokes of a character, such as the vertical strokes 
17102 shown in FIG. 171 to have its leftmost edge slightly into one subpixel column, with total 
coverage in three successive subpixel column to the right, so as to cause the second leftmost pixel 
column 17103 in the resulting bitmap to have one or more pixels totally covered so as to require no 
color-balance spreading. 

In such an optimized hinting process, subsequent vertical strokes would be aligned to 
cover three adjacent subpixel column starting at a distance of three, six, or nine subpixel column 
from the rightmost edge of the first vertical stroke. This would cause the subsequent vertical 
strokes, such as the vertical strokes 17104 and 17106 shown figure 171, to have multiple pixels 
that are totally covered, so as to require no color balancing. 

Although fonts of the type shown in FIG. 168 to 171 are more readable than most subpixel- 
optimized font bitmaps produced by prior art method, as a result of these investigations the 
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inventor has figured how to produce even more clear subpixel optimized font, as shown in FIGS. 
172 through 174. 

FIG. 172 shows a whole-pixel grayscale bitmap representing a subpixel-optimized 320 by 
240 pixel display of a web page of the type shown in FIG. 168, except that it uses a new, more 
clear method for producing and displaying font bitmaps. 

FIG. 173 shows a four times blowup of the portion of FIG. 172 shown in the dotted box 
numbered 17200.. 

FIG. 174 shows a further four times blowup of the portion of text shown in the dotted lines 
17300 in FIG. 173. 

As can be seen from looking at FIGS. 172 through 174, there is relatively little horizontal 
spreading of color values from many of the vertical strokes contained in the font bitmaps shown in 
those figures. It should be noted that the uniform light gray background in FIGS. 173 and 174 
results because the text in those figures was taken from a portion of the web page of FIG. 172 that 
had a background color, not because of any spreading due to color balancing. The fonts shown in 
these figures are substantially more clear than those shown in FIGS. 168 and 169. 

The inventor has made this improvement by aligning the leftmost edge of a character's 
leftmost vertical stroke with the left edge of a pixel boundary. In many embodiments this is done by 
inserting three padding subpixel column 17500, shown in FIG. 175 before the leftmost subpixel 
column that contains a non-zero coverage values. This automatically aligns the leftmost 
rasterization unit (i.e., subpixel) all or partially covered by a character's outline with the leftmost 
edge of a pixel column. If a characters is hinted so its leftmost outline edge is aligned with the 
leftmost edge of a rasterization unit, this will automatically cause that leftmost outline edge to be 
aligned with the leftmost edge of a pixel in the resulting font bitmap. When the leftmost edge of a 
font outline is a vertical stroke this makes it very easy to create a font bitmap that has clear leftmost 
vertical edge, even after non-linear color balancing. 
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FIG. 176 shows one of many possible hinting interfaces that can be used with the present 
invention. In this hinting interface the dotted lines 17602 is a line that can be moved by the user to 
interactively define the left side bearing for a desired character. The dotted line 17604 is a 
movable line that defines the right side bearing. The left side bearing is the distance between the 
initial reference point, sometimes called the pen position, relative to which a character is to be 
drawn and the leftmost edge of the bitmap of the character being drawn. The line 1 7604 
corresponds to the location relative to the bitmap at which the pen position will normally be placed 
at the start of the drawing of the next successive character along a line of text. The right side 
bearing is the distance between the line 17604 and the rightmost edge of the bitmap of the 
character being drawn. The advance width is defined as the distance between the lines 17604 and 
17602. This represents the normal total width between pen positions before and after the drawing 
of a character's bitmap. In some embodiments, the left side bearing value and the advance width 
are rounded to whole multiples of pixel widths, although in other embodiments this need not be 
true. In some cases, the left and/or right side bearing values can be negative. For example this 
often happens with italic fonts in which the bitmaps associated with successive characters often 
overlaps portions of each other's advance width. 

Each of the small rectangular dots 17606 shown in FIG. 176 correspond to the center of a 
rasterization unit, which, in subpixel-optimized font bitmaps, correspond to an individual subpixel. 
In this particular hinting interface rasterization units more than half covered by a character-font 
shape's outline are shown in black, although in more advanced interfaces such rasterization units 
could be shown with grayscale coverage values. The character-font shape's outlines are shown in 
the figure and each point that defines a segment in the outline is numbered, whether it be a control 
point or a segment endpoint. 

FIGS. 177 through 181 are used to help explain some of the steps described in the highly 
simplified pseudocode contained in FIG. 182. 

FIG. 182 is a highly simplified pseudocode description of programming 6000A, which 
corresponds generally to the pseudocode shown in FIG. 60, except that the pseudocode shown in 
FIG. 182 focuses on computational aspects that relate to the improved method of producing more 
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clear non-linearly color-balanced subpixel-optimized bitmaps described above with regard to FIGS. 
172 through 175. 

The pseudocode includes a function 18202, which determines the tightest rectangular 
array of rasterization units into which a character-font shapes can be placed, taking into account 
the alignment of its shape relative to such rasterization units defined by its hinting. 

The position of the font outline relative to the individual rasterization units in which it occurs 
is not changed by this function. Thus, if such an outline's leftmost point occurs other than at the 
left edge of the rasterization unit it is in, that rasterization unit will appear at the leftmost edge of the 
tightest rectangular array produced by function 18202, and the leftmost point of that outline would 
occur within the leftmost rasterization unit column of that rectangle, but it would not occur at the 
leftmost edge of that leftmost column. 

FIGS. 177 and 178 are used to help explain this function. FIG. 177 corresponds to a 
hinted character-font shape outline. FIG. 178 shows the rectangle of rasterization units (each 
corresponding in size to a subpixel) returned by function 18202 for the character outline shown in 
FIG. 177. This grid corresponds to the tightest, or smallest, rectangle into which the rasterization 
units containing the character font shape fits. 

Once the function 18202 has been completed, functions 6002A through 6006 are 
performed. These correspond to steps 6002 through 6006 of FIG. 60. They are used to determine 
a coverage value for each rasterization unit contained in the rectangle returned by function 18202. 
Each such coverage value represents the percent of the subpixel covered by the higher resolution 
character-font shape outline being rasterized. 

FIG. 179 illustrates the coverage values calculated for each rasterization unit in the array 
shown in FIG. 178. In it coverage is represented by the percent of the rasterization unit that is 
colored black. In FIG. 179 the portion of the resulting bar graph in each rasterization unit 
representing coverage is placed at the top of that unit if the corresponding part of the unit covered 
by the character-font shape outline occurs at the top of the rasterization unit. 
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In FIG. 180 the bar graphs for all individual rasterization units are placed starting at the 
bottom of the corresponding subpixel unit, so as to make them correspond more closely with the 
representation of coverage values shown in FIGS. 46 through 52 and 92 through 93, described 
above. 

Once the character-font shape has been rasterized, step 18204 maps the resulting array of 
subpixel coverage values into an array of subpixel-addressable pixels. It does so aligning the first 
column of rasterization units in the tight rectangle described above with the leftmost subpixel of a 
pixel row. This causes the leftmost column of rasterization units that have a nonzero coverage 
value to be placed as a leftmost subpixel column in a whole pixel as described above with regard 
to FIG. 175. In the example illustrated in FIGS. 177 through 181 , this causes the resulting subpixel 
array to appear as shown in the central five pixel columns labeled 18102 in FIG. 181 . 

Next a step 18206 pads the bitmap array being created for the current character with a 
pixel column 1804 comprised of three subpixels. This padding column is placed to the left of the 
pixel having the leftmost subpixel column containing an actual nonzero coverage value. This 
causes the subpixel array in the example to appear as shown by the combination of pixel columns 
18104 and 18102 in FIG. 181. 

Next a step 18208 pads the bitmap array with two or more subpixel columns to its right, so 
as to cause the total number of subpixel columns of the bitmap to be an even multiple of three, that 
is, to be an even number of whole pixel columns. This causes the example subpixel array to 
appear as shown by the combination of pixel columns 18104, 18102, and 18106 in FIG. 181. 

Step 18210 adjusts the left and right side bearing value to compensate for the addition of 
the padding pixel columns. Thus, for example, a bitmap that would otherwise have a left side 
bearing of one pixel width would be changed to have a left side bearing of zero to compensate for 
the addition of the left side padding column. Similarly a bitmap that had and extra pixel column 
added to its right side would decrease its right side bearing by one pixel width. 
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Next function 18212 performs non-linear color balancing, which in many embodiments will 
correspond to the steps described by the loop 6008 shown in FIG. 60, described above. 

Once this has been done, in embodiments using a packed color value representation of 
the type described in FIG. 96 above, step 18214 converts the pixel color values resulting after the 
color balancing operation into corresponding values from a more limited color palette. 

Note that the method of FIG. 182 allows room for any color balancing that might be 
necessary, without tending to cause the unnecessary color spreading discussed above with regard 
to FIGS. 168 and 169. It does this by insuring that there are at least two subpixels to the left and to 
the right of any subpixels corresponding to area covered by the font shape being rasterized. 

In other embodiments of this aspect of the invention other methods will be used to cause 
leftmost and rightmost edges of font shapes and vertical strokes to be aligned with whole pixel 
boundaries, so as to take maximum advantage of the capability of non-linear color balancing to 
reduce smearing. In some such embodiments, whether or not padding pixel columns were added 
to the left or right side of a font bitmap could be a function of whether or not color balancing 
distributions were required in such locations. 

FIG. 183 describes functions for drawing a string of characters using the bitmaps produced 
by the method described in FIG. 182. This pseudocode is similar to that described above with 
regard FIG. 97, except that it focuses on aspects of the invention used with bitmaps produced by 
the method of FIG. 182. 

When the draw string function 18300 shown in FIG. 183 is called, a step 18302 sets the 
pen position to a start position- specified by the draw string call that indicates where the display of 
the string is the start. 

Then a loop 9714A similar to the loop 9714 described in FIG. 97 is performed for each 
character the string to the display. 
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In this loop a step 9716 accesses the current character's font bitmap. Then a step 18304 
sets the character start position to the current pen position. Then a step 18306 adjusts the current 
pen position by the left side bearing. As has been described above, the left side bearing has been 
changed from what it would normally be to take into account the fact that the character bitmap has 
been padded with one extra pixel column on its left hand side, and thus the left side bearing has 
been decreased by the width of one pixel column. 

Next a step 971 8A is performed for each pixel in the font bitmap. This includes a substep 
18308, which tests to see if the current pixel's value is nonzero. If so, it draws the pixel on the 
screen at a position determined as a function of the current pen position. 

If the current pixel's value is zero, it represents a totally transparent pixel, meaning the 
background color previously at the position of the current pixel should be left unchanged. In this 
embodiment of the invention the functions described in FIG. 96 reserve the value 0 to represent 
such a totally transparent pixel. 

This practice of not writing transparent pixel's is applied to all pixels of the bitmap in the 
embodiment described in FIG. 183. This practice is particular valuable with regard to pixels in the 
padding column placed at the left most edge of a character-font bitmap by step 18206 described 
above regard FIG. 182. This is because pixels in such padding columns will commonly have no 
color values spread into them as a result of non-linear color-balancing when vertical stroke 
boundaries have been aligned to vertical pixel boundaries. As a result, such pixels will be 
transparent and color values that may have been placed in their location by the character to its left 
can remain unchanged, allowing the pixel columns of adjacent characters that contain coverage or 
color balancing information to be placed adjacent to each other. 

This can be seen for example at the location indicated by the numeral 17302 in FIG. 173 
where the pixel column between the V and "e" of the word "Web" contain color values from the 
"w" that have been allowed to show through the transparent, and thus non-written, left side padding 
column associated with the "e". This can also be seen at the location indicated by the numeral 
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17402 shown in FIG. 174, in which the pixel column between the Y and the "e" contain color 
values from the V that are not overridden by the transparent padding pixel column of the V. 

As those skilled in the art will recognize, function 971 8A will require some sort of iteration 
controlling the position at which pixels are drawn to be repeated for each row of a font bitmap, so 
as to have each of its bitmaps drawn in the proper place. 

It should be appreciated that in other embodiments of the invention, functions could be 
provided that would allow overlapping non-transparent pixel values from adjacent characters to be 
combined, rather than merely allowing non-transparent color values from one character to show 
through when the corresponding pixels of the following character are transparent. 

Preferably such a process would allow combination of such transparency values on a 
subpixel-by-subpixel basis. Such a process could provide an even more accurate representation 
of closely spaced letters, although it would require more computation. 

One way of achieving this result would be as follows: Add each of the three corresponding 
alpha component values associated with any overlapping pixel between characters, clipping any 
component values at their maximum possible value. And then drawing each of the resulting pixels, 
using the combined component alpha values to determine how much foreground color and how 
much background color should be drawn at its location. 

FIG. 184 illustrates a hinting interface similar to that described above regard FIG. 176, 
except that it includes an interface feature 18402 comprised of a user-movable line or control. This 
control allows the user to selectively position, relative to his or her character-font shape outline, the 
location to be aligned with the leftmost edge of a pixel column following the leftmost padding pixel 
column. 

Such an interface feature is more desirable when hinting fonts that have a leftmost edge 
that is other than a single vertical stroke. For example, when dealing with a character-font shape 
having a leftmost main vertical stroke with a small serif sticking out from to its left edge by less than 
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a full pixel width, the hinter may want to have the main leftmost edge of the main vertical stroke 
aligned with a whole pixel boundary, rather than the more leftward serif. The interface feature 
shown in FIG. 184 would make such an alignment easy for a hinter to select. 

Another way of giving a hinter the equivalent capability would be to allow him or her to 
select whether to add only two subpixel padding columns, as described above with regard FIG. 170 
or 171, or to add three or more such subpixel padding columns, as is described above with regard 
to figs. 175, 181 and 182. 

The just described method for making non-linear color balanced subpixel optimized 
bitmaps more clear is not only applicable to small fonts of the types shown in figs. 172 through 174 
but also to larger fonts, such as the relatively large font shown in FIG. 55. 

It should be appreciated that subpixel optimization can usually represent a font bitmap with 
just three different types of pixels: foreground pixels, background pixels, and intermediary, color- 
balancing, pixels. A foreground pixel represents a portion of the font image totally covered by the 
font shape being represented, and is drawn with the foreground color with which the character is 
being represented. A background pixel represents a portion of the font image totally uncovered by 
the font shape, and is drawn with the color of the background on top of which the font is being 
shown. An intermediate pixel represents a pixel that is partially covered by the font shape and/or 
that receives color balancing distributions for a nearby pixel that is partially covered. The color of 
each of its subpixels is determined separately by color balancing. 

When prior art linear color balancing of the type described above with regard to FIGS. 46, 
47, 52, and 93 are applied to fonts, color balancing is performed across every edge of a character 
shape in the direction of subpixel color variation, even if that edge is perfectly aligned with a pixel 
boundary. This leads to spatial smearing of the shape of all letters, no matter how well hinted. 

When non-linear color balancing of the type described above with regard to FIGS. 48, 49, 
51 , and 91 is applied to fonts, hinting can be used to greatly reduce the spatial smearing caused by 
color balancing. In portions of a character's shape where its edges are aligned with pixel 
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boundaries, often no color balance distribution will be required across pixel boundaries. This is 
because such non-linear color balancing only distributes color imbalance that occurs within a give 
pixel. This allows foreground pixels to be next to background pixels along the direction of subpixel 
color variation in such locations, greatly increasing the perceived clarity of the font shape. This is 
shown in FIG. 173 and 174 in which substantial portions of the vertical strokes in the 8 pixel per em 
font shown those figures have been hinted so that the edges of those strokes align with pixel 
boundaries. As a result, foreground pixels are located horizontally next to background pixels along 
substantial portions of the edges of many such vertical strokes. Even with the less optimal hinting 
of leftmost vertical stroke edges shown in FIGS. 168 and 169, the amount of color-balance 
smearing is substantially less than that which would result from prior art linear color balancing. 

FIGS. 185 through 190 are highly simplified pseudocode descriptions of user interface 
innovations that can be used to improve the browsing of Web pages, particularly when such 
browsing is performed on relatively small or relatively low resolution screens. 

FIG. 185 is a higher level description of the selected-text reflow method described above 
with regard to FIGS. 129 through 134. This method 18500 includes a function 18502 that 
accesses a Web page's content and a function 18504 that performs a first layout of the Web page's 
content, placing text at different horizontal locations indicated for text in the web page. 

The markup languages used to describe Web pages have multiple methods of indicating 
that different portions of text are to be drawn at different horizontal locations or in different 
horizontal ranges in a web page, including, to name just two, the use of tables and frames. 

Once such a layout has been performed, function 18506 displays the elements of the 
layout at a given scale and at relative positions determined by the first layout. After this display has 
been performed a step 18508 enables the user to select a portion of the text at a given horizontal 
location in the display of the first layout. On way of enabling this is described above with regard to 
FIG. 130. 

If such a selection is made, function 18510 causes function 18512 and 18514 to be 
performed. Function 18512 performs a second layout of the text that has been selected by the 
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user. This second layout re-flows the selected text across the lines of the new column in which the 
text has a different, usually larger, font size relative to the width of the lines in that column. When 
this second layout is been performed, function 18514 displays the layout of the new column at a 
scale that fills at least two thirds of the width of the screen or screen window on which the web 
page is being displayed. 

As indicated above with regard to FIGS. 135 through 137, the second layout in such a 
selected-text re-flow method allows a user to see selected portions of a Web page's layout in at 
large easy read font sizes. This can be a tremendous advantage on both low resolution screens, 
screens that are small, and/or screens that are relatively far from their viewer. The first layout in 
such a method allows the user to get a view of how the web page is intended to look in more 
normal displays, and allows the user to more rapidly select which portions of the text he or she 
desires to see re-displayed at a larger font size. 

FIG. 186 is a high-level pseudocode description of a zoom-to-fit method 18600, of the 
general type described above with regard to FIGS. 118 through 120. 

This method includes a function 18602 that accesses a Web page's content, and a 
function 18604 that lays the Web page's content out. 

Once such a display of the layout is being shown on a screen, function 18608 enables the 
user to drag a pointing device across this display. During such a drag, if the drag continues across 
a boundary associated with a screen edge, a function 18610 causes function 18612 to scroll, onto 
the screen, portions of the layout that were previously off screen on the other side of the screen 
edge. This is done to allow users to select with a drag a portion of the layout that is either too large 
to entirely fit on the screen at the current display scale or that was positioned at the start of a drag 
so that only part of it was on the screen. 

If the user releases the drag, function 18614 causes functions 18616 and 18618 to be 
performed. The first of these causes a part of the layout to be defined as having been selected 
based on the positions in the layout that corresponds to the start and end of the drag. Such a 
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selected part can correspond to a portion of the layout having either the horizontal or vertical range 
of the drag or having an area having diagonal corners corresponding to the start and end of such a 
drag. Then function 18618 displays the selected part of the layout at a scale that causes it to 
substantially fit the screen. By substantially fit the screen we mean, that the selected width, height, 
or area fills at least two thirds of the corresponding dimension or dimensions of the screen, or 
portion of the screen in which the display is being made. 

FIG. 187 is a high-level pseudocode description of a drag scroll method 18700 that allows 
a user to easily navigate within the display of a web page's layout. 

This method includes a function 18702 that accesses the Web page's content, a function 
18704 that performs a layout of the Web page's content, and a function 18706 that displays all or 
portion of that layout at a given scale factor. Then a function 18708 enables the user to drag a 
pointing device across the display of the layout. Function 18710 responds to any such drag across 
a boundary associated with a screen edge by scrolling onto the screen, past the screen edge, 
portions of the layout previously off screen. 

This method can be used as part of, or independently from, zoom selection functions. It 
has the advantage of enabling a user to scroll around the display of the layout of a web page by 
merely dragging a pointing device across a boundary at, or near, an edge of the display screen. 
This is substantially more natural and faster than scrolling by means of scroll bars. 

FIG. 188 is a high-level pseudocode description of a click-zoom method 18800 that 
enables the user to rapidly select to zoom in on a desired portion of the display of a layout of a web 
page. This method includes a function 18802 that accesses the web page's content, a function 
18804 that performs a layout of the Web page's content, and a function 18806 that displays all or a 
portion of the Web page's layout at a first scale. A function 18808 enables the user to click a 
pointing device at a selected location in the display of the layout shown at the first scale, and 
function 18810 responds to such a click by performing a zoomed-in display of the portion of the 
layout around the location in the layout at which the click was performed. Commonly the zoomed- 
in display will be centered on the location in the layout at which the click was made. 
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FIG. 189 is a highly simplified pseudocode description of the zoomclick method 18900 
described above in some detail with regard to FIGS. 121 through 128. 

This method includes a function 18902 that accesses a Web page's content, a function 
18904 that performs a layout of that content, and a function 18906 that displays all or a portion of 
the web page's layout at a first scale on a display screen having an associated pointing device. In 
the particular embodiment of this method described in FIG. 189, the screen is a touch screen and it 
is intended that the pointing device can be a person's finger. 

Once the display of the layout at the first scale has been performed, a function 18908 
responds when a press has been made to the touch screen display. When such a press occurs, 
this function causes functions 18910 through 18922 to be performed. 

Function 18910 replaces, on the screen, the display of a portion of the web page at the 
first scale with a zoomed-in display of a portion of the web page at a larger scale. This zoomed 
portion includes a selected location in the layout associated with touch screen press. Preferably 
the selected layout position has substantially the same location on the screen in the zoomed-in 
display as it had in the display at the first scale at the time of selection. By substantially same 
position, it is meant that the selected positioned should have a location on the screen both 
immediately before and after the zoom that appears to correspond to the same touch positioned on 
the screen. Preferably this would mean that the change in the selected position's screen location 
would not change by more than twenty percent of the width or height of the screen immediately 
after such a zoom (and preferably by considerably less than that amount). 

Once the zoomed-in display is shown, function 18912 displays a cursor above the location 
at which the screen is being touched to indicate the selected location in the web page's layout that 
is associated with the touch. In some touch screen devices, particularly those designed for use 
with styluses having relatively fine points, there is no need for such a cursor, since the user can 
see with considerable accuracy the point at which the screen is being touched. But in touch 
screens designed for use with fingers as pointing devices it is often desirable to place a cursor 
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above the location at which the screen is being touched so the user can accurately see the location 
in the screen's display that is associated with such a touch. This is particularly desirable when the 
method is being use with a display, such as that shown in FIGS. 121 through 128, that is relatively 
small compared to the size of a human finger. 

During the continuation of the touch a function 18914 responds to any movement of the 
touch by correspondingly moving the cursor in the zoomed display. Also during the continuation of 
such a touch, a function 18916 response to any movement of the touch across a boundary 
associated with a screen edge by scrolling onto the screen, past the screen edge, portions of the 
layout at the zoomed scale that were previously off the screen. This allows the user to rapidly and 
conveniently scroll within the zoomed display of the web page while in zoomclick mode. 

Function 18918 responds if the user releases a touch at a given positioned in the zoomed 
display of the web page. If so, a function 18920 acts as if a pointing device click had occurred at a 
positioned in the web page corresponding to that of the release. For example, if the release is at a 
layout location corresponding to a web link, the system will respond by selecting the link, or if the 
release is at the location of a radio button, the system will respond by flipping the state of the radio 
button. 

Once this has been done, a function 18922 replaces the display of the zoomed-in layout 
on the screen with a display of the layout at the same first scale factor at which the web page was 
displayed before the pointing device press was detected by function 18908. 

As described above with regard to FIGS. 121 through 128, zoomclick provides a valuable 
technique for allowing a user to rapidly see and select desired portions of a web page at a zoomed- 
in scale that makes the contents of those selected parts easier to read and easier to accurately 
select with a pointing device. 

FIG. 190 is a highly simplified pseudocode description of a method 19000 that allows a 
user to see a zoom-out view of a web page using greeking to represent text lines. Greeking is the 
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representation of the size at which portions of text are laid out in a document by non-readable 
graphic representations. 

This method includes a function 19002 that accesses a Web page's content, a function 
19004 that performs a layout of the web page's content, and functions 19006 and 19014 that 
detects the scale at which the user has selected to have the layout of the web page's contents 
display. 

If the user has selected to have the web page's layout displayed at a given larger display 
scale, function 19006 causes function 19008 to display a portion of the web page's layout at the 
larger scale. This includes performing a function 19010 to represent the layout's images with 
bitmap images scaled for display at the larger scale and a function 19012 that represents the 
layout of the web page's strings with bitmaps composed from separate font bitmaps that have sizes 
appropriate for display at the larger scale. 

If, on the other hand, the user has selected a given smaller display scale, one which is so 
small that at least some of the text of the web page cannot be displayed at that scale in a size that 
is readable, function 19014 causes a function 19016 to display a portion of the web page's layout 
at the smaller scale. This includes performing a function 19018 that represents the layout's images 
with bitmap images that have been scaled down for display at the smaller scale, and a function 
19020 that represents at least some strings with bitmaps composed of greeked text 
representations that indicate the size and location of individual strings in the display at the smaller 
scale. 

In many cases the bitmaps used to represents strings in such greeking will merely be lines 
or rectangles having a width and/or height corresponding to the size of their corresponding strings 
in the web page's layout at the small-scale. 

When a layout is displayed at a size in which text is too small to read the use of greeked 
representations of text can makes such a display easier and more pleasant to see, and such 
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greeking generally takes less computation to generate than would corresponding string images 
generated from unreadably small font bitmaps. 

One of the major uses of the method shown in FIG. 190 is to enable a user to quickly gain 
an overview of a web page's layout and to allow him or her to quickly select different portions of 
such a web page, such as has been described above with regard to FIGS. 136 and 137. 

Those skilled in the art of computer user interfaces will appreciate that some of the 
methods described in FIGS. 185 through 190 can be used in combination with each other and with 
other aspects of the invention described above as part of a single user interface mode, whereas 
others are them would normally be used in different user interfaces or different user interface 
modes. 

FIGS. 191 through 203 illustrate an aspect of the invention that allows a user to have a 
web page displayed at a first and second scale views. In the first scale view images, column 
widths, and horizontal displacements of images and columns have a first set of sizes. In the 
second scale view such images, column widths, and horizontal displacements have a second, 
smaller, set of sizes, but the size of fonts relative to images, column widths, and such horizontal 
displacements are larger in the second scale view than in the first. 

The embodiment shown in these figures also allows a third scale view to be created in 
which images, column widths, horizontal displacements, and fonts all are displayed with sizes 
larger than in either the first and second scale views. 

The embodiment of FIGS. 191 through 203 shows these different scale views on a fixed 
resolution screen 19302 of a personal digital assistant, or PDA, 19304, shown in FIG. 193. In this 
embodiment the screen has a resolution of 320 x 240. The first scale view displays a web page 
with a virtual resolution of 640 by 480. This means that the first scale view displays images, 
column widths, and horizontal displacements with the same sizes relative to the 320 by 240 screen 
as those same elements would have relative to a 640 by 480 screen if they were displayed at that 
larger resolution. The second scale view displays the web page at a virtual resolution of 800 by 
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600. This means it shows images, column widths, and horizontal displacements with same sizes 
relative to the 320 by 240 screen as they would have if the web page were being displayed on an 
800 x 600 resolution screen, but in the second scale view the fonts have a larger size relative to 
images, column widths, and horizontal displacements than in the first scale view. 

The fonts are larger relative to other features in the second scale view so that they can be 
actually displayed on the screen 19302 with the same sized font bitmaps as are used in the first 
scale view. This is done because the fonts used for most text in the first scale view are 8 pixel per 
em fonts, which are about as small as can be easily read. The first scale view scales down its 
virtual 640 by 480 resolution by a factor of two to make it fit on the 320 x 240 screen. But the 800 x 
600 layout in the second scaled view is scaled down by even more, by two and a half times, to 
make it fit on the 340 x 240 screen. Thus, to make the font bitmaps for similar text in the second 
scale view also be 8 pixels per em, the relative font sizes have to be approximately twenty-five 
percent larger in the layout of the second scale view than in the layout of the first scale view. This 
use of the same displayed font sizes in the second scale view enables text to remain easily 
readable despite the large degree to which a web page's layout is scaled down in that view. 

As stated above, in the embodiment shown in FIGS. 193 through 203 a third scale view 
can also be selected. This view shows image sizes, column widths, and horizontal displacements 
with the same size relative to the screen as if the web page were being shown on a 480 x 320 
screen, but with substantially the same font size relative to images, column widths, and horizontal 
displacements as in the first scale view. This causes larger font bitmaps to be used for the display 
of the same text in the third scale view as in the first scale view. 

FIG. 191 is a highly simplified pseudocode illustration of one method for achieving this 
aspect of the invention in conjunction with the embodiment of the invention described above with 
regard to FIGS. 102 through 109. 

When the user is viewing a web page on the PDA 19304 shown in FIG. 193, he or she can 
select to display a pop-up menu by pressing a specified one of the PDA's buttons 19302. 
If the user selects to display such a pop-up menu, function 19100 of FIG. 191 causes 
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functions 19101 through 19140 to be performed. Function 19101 displays a pop-up menu 
19402, shown in the dotted lines 19402 in FIG. 194. 

Once function 19101 has displayed the pop-up menu, if the user selects the display size button 
19404 shown in FIG. 194, function 19102 of FIG. 1 causes function 19104 to display the 
size option menu. In the embodiment shown, this menu is comprised of the three buttons 
19502, 19504 and 19506, shown in FIG. 195. By selecting one of these buttons the user 
can select the set of parameters, including virtual resolution, font size, and scale factor, 
associated with each of the first, second, and third scale views as a group, without having 
to select two or more of those parameters separately. 

The user can select a display of the. second scale view (the 800 x 600 virtual resolution 
view) by pressing the button 19502 shown in FIG. 195. If the user presses this button, function 
19106 causes functions 19108 through 19116 to be performed. 

Function 19108 causes the display of the pop-up menu 19402 and the display size option 
menu to be removed from the display screen. Function 191 10 sends a virtual resolution command 
to the proxy server 210 shown in FIG. 102. 

The generation of the virtual resolution command includes sub-functions 191 12 through 
191 16. Function 19112 requests that the set of fonts to be used in the display at the 800 by 600 
virtual resolution be a set of smaller fonts that includes 8, 9, and 1 0 pixel per em fonts. This 
corresponds to font sizes that would have point sizes of 8, 9, and 10 points, respectively, on a 
computer such as a Macintosh that defines point size in terms of 72 dpi. On computers, such as 
IBM PC compatible computers, that define point size based on 96 dpi, these point sizes would be 
approximately 2 A as large, or 6, 7, and 8 points, respectively. 

In the preferred embodiment a high percent of most web page's text will be displayed with 
the 8 pixel per em fonts. For such small fonts it is desirable to use anti-aliased fonts. In a 
preferred embodiment non-linear color balanced 8 pixel per em fonts, such as those shown in 
FIGS. 172 through 174 are used. These same fonts are shown in detail FIGS. 204 through 209. 
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FIG. 204 illustrates a paragraph of text shown in FIG. 202 blown up four times. FIG. 205 
illustrates the word "hydrogen" shown surrounded by dotted lines in FIG. 204 blown up again by 
four times. FIGS. 206 through 209 separately illustrate the "h", "y", "d", and "e" from the word 
"hydrogen" shown in FIG. 205. 

In FIG. 204 the first line of text corresponds to heading text and is shown with a relatively large 
10 pixel per em font. The remaining text is body text and is displayed with the more 
common 8 pixel per em font. 

As can be seen from FIGS. 204 through 209, in these fonts a majority of characters whose 
shape has two vertical strokes that occur at the same height relative to the character's shape have 
at least one pixel row in which relatively covered pixels representing the two vertical strokes are 
separated by at least one pixel that is relatively uncovered. In each FIGS. 206 through 209 two or 
more relatively covered pixels are labeled 20602 and a relatively uncovered pixels is labeled 
20604. The relatively covered pixels are ones whose areas are more covered by the shape of the 
character being represented than is the area of the relatively uncovered pixel. In a font display, 
such as that in FIGS. 204 through 209, where a character's foreground color is dark and its 
background color is light, the relatively covered pixels will be relatively dark and the relatively 
uncovered pixel will be relatively light. 

Also in these fonts, as can be seen from FIGS. 204 and 205, the majority of lower case 
letters that do have two such horizontally spaced vertical strokes also have an advance width - 
that is, a total number of pixel columns corresponding to the width of a character, including any 
inter-character spacing - of 4 pixel columns or less. This is a very small advance width for such 
relatively easy to read characters. 

In FIG. 205 the advance width of each of the letters of "hydrogen" is shown with a number. 
As can be seen from that figure, all of the word's letters except "r" have an advance width of four, 
and "r", which does not have two vertical strokes, has an advance width of 3. 
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Returning to FIG. 191 , function 191 14 requests that the virtual screen size to be used in 
the layout performed by the proxy server for the second scale view be 800 pixels by 600 pixels. 

Function 191 16 sets the scale factor for the second scale view to 2.5. This scale factor will 
cause the string measurements returned by function 10618 of FIG. 106A for use in the proxy 
server's virtual layout to be scaled up 2.5 times from the 8, 9, or 10 pixel per em size of the smaller 
font set used in the second scale view. If this scaling up would result in a string measurement that 
included a fractional pixel size, function 10618 will round up the string measurement it returns to 
the proxy server's browser software to the next whole pixel size. The 2.5 scale factor will also 
cause the actual positions and sizes of features in the virtual layout created by the proxy server's 
browser software to be scaled down by factor of 2.5 when entered into the download display list 
1 021 2 shown in FIG. 1 02, for display on the screen of the client computer 200 shown that FIG. 
102, which corresponds to the PDA 19304 shown in FIG. 193. 

FIG. 192 shows how the pseudocode functions numbered 10554 through 10560 in FIG. 
105B can be modified to better respond to the type of virtual resolution commands generated by 
functions 19110, 19122, and 19134 of FIG. 191. This portion of pseudocode is identical the 
corresponding portion of FIG. 105B except for the addition of the lines shown underlined in FIG. 
192. 

In FIG. 192 the underlined functions 19202 and 19204 have been added so that the proxy 
server computer's response to a virtual resolution commands includes not only changing the 
browser's virtual screen resolution to that requested in the command, but also changing the font 
set and scale factor used by the proxy server in its creation of the virtual layout 10206 and 
corresponding scaled-down download display list 10212 that are described above with regard to 
FIG. 102. In the pseudocode shown in FIG. 192 all three of these changes are made before the 
function 10560 makes a call to the screen redraw function of the proxy server's browser software 
that will cause the current web page to be re-laid out at the new virtual screen resolution using the 
newly requested font set and scale factor. 
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The user can select the 640 by 480 virtual resolution layout of the first scale view by 
pressing the button 19504 shown in FIG. 195. If the user does so, function 19818 of FIG. 191 
causes functions 19120 through 19128 to be performed. These functions correspond exactly to 
functions 19108 through 191 16 described above with regard to the 800 by 600 virtual layout, 
except that: (1) function 19124 sets the requested virtual screen size to 640 by 480 pixels, rather 
than 800 pixels by 600 pixels; and (2) function 19128 sets the scale factor to 2, rather than 2.5. 

If the 640 x 480 virtual screen resolution view is selected, the string measurements 
returned by function 10618 of FIG. 106A for use in the proxy server's virtual layout will be scaled up 
from the 8, 9, or 10 pixel per em size of the requested smaller font set by a factor of 2. This will 
cause the fonts to have a relatively smaller size relative to elements such as images, column 
widths, and horizontal displacements in the first scale view than in the second, 800 x 600 virtual 
resolution scale view. 

If the user selects the third scale view, i.e., the 480 by 320 virtual resolution view, by 
pressing button 19506 of FIG. 195, function 19130 of FIG. 191 causes functions 19132 through 
19140 to be performed. These functions are identical to functions 19108 through 191 16 described 
above with regard to the 800 by 600 layout except that: (1) function 19136 requests a virtual screen 
size of 480 by 320 pixels; (2) function 19138 requests that the font set used by the proxy server be 
a larger font set than that used in either the 800 by 600 or the 640 by 480 virtual resolution 
displays; and (3) function 19140 sets the scale factor to 1.5. 

The ratio of the font sizes in the large font set used in the 480 by 320 virtual resolution 
display to the font sizes in the smaller font set used in the 640 by 480 virtual layout is roughly 4/3. 
This is the inverse of the ratio of the scale factor of 1 .5 used in the 480 by 320 virtual layout relative 
to the scale factor of 2 used in the 640 by 480 virtual layout. As a result of this relationship the 
fonts in the 480 by 320 virtual resolution view have approximately the same size relative to images, 
column widths, and horizontal displacements as do the fonts in the 640 by 480 virtual layout. The 
actual size of the fonts displayed on the 320 x 240 screen in the 480 by 320 virtual resolution 
display are 1 1 , 1 2, and 1 3 pixels per em. 



BIT02-2NP Assignee: Bitstream Inc Contact Phone No: 61 7494-1 722 or 61 7 497-6222 Page 1 52 



FIGS. 196, 197 and 198 illustrate views of the same web page when laid out, respectively, 
at the 480 by 320, 640 by 480, and 800 by 600 virtual resolutions described above with regard to 
FIG. 191 . All of these figures are screen shots taken from a 320 by 240 pixels screen that has 
been generated on a client computer from a display list similar to the display list 10212 created by 
the proxy server 210 shown in FIG. 102. 

As can be seen by comparing FIGS. 196, 197, and 198, the size of the images 19602 and 
19604, the width of the columns 19606 and 19608, and the horizontal displacement of those 
columns gets smaller as the virtual layout resolution and the scale factor increase. As can also be 
seen by comparing FIGS. 197 and 198, the size of the fonts relative to the size of such images, 
column widths, and column displacements is larger in FIG. 198 than in FIG. 197, even though the 
actual size of the fonts is the same in both figures. 

As has been stated above, it is common today for many web pages to be laid out with a 
specified layout width of 800 pixel. A comparison of FIGS. 196, 197, and 198 illustrates that the 
800 by 600 virtual resolution display of FIG. 198 show a higher percent of the width at which its 
web page is intended to be viewed than do the other views. Thus, it provides a better view of the 
web page. 

Note that where a vertical scroll bar is required on the display of the 320 x 240 screen at 
any of the three different scale views, the actual width of the virtual resolution layout will be 
decreased by the width of the vertical scroll relative to the virtual layout. As can be seen by 
comparing FIGS. 196 and 197, the width of the scroll bar has been scaled down in the 640 x 480 
resolution layout relative to its width in the 480 x320 resolution layout by the ratio of the 2.0 and 1 .5 
scale factors that are used, respectively, in those different views. In the 800 x 600 virtual resolution 
view shown in FIG. 198 the displayed vertical scroll bar has the same width as in the 640 x 480 
virtual resolution view, to prevent it from becoming so narrow that it would be difficult for users to 
easily operate it. 

FIG. 199 is identical to FIG. 197, except that is shown on the same drawing sheet as FIG. 
200, which shows the display of the same web page as it would appear on an actual 800 by 600 
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pixel screen generated by a normal browser program (as scaled down to make FIG. 200 the same 
size as FIG. 199). Because of the way in which the particular web page shown in FIGS. 199 and 
200 specifies the size of various layout elements, the horizontal width of non textual elements 
varies more between the scaled-down virtual 800 by 600 view of FIG. 199 and the actual 800 by 
600 pixel display than it would for most web pages. 

FIGS. 201, 202, and 204 illustrate views of another web page. 

FIG. 201 illustrates a view of this other web page generated on the screen 19302 of the 
PDA 19304 shown in FIG. 193 when that computer is operating in the first scale, or 640 by 480 
virtual resolution, view described above with regard to FIG. 191. 

FIG. 202 is similar to that in FIG. 201 except that it is generated using the second scale, or 
800 by 600 virtual resolution, view described above with regard to FIG. 191 . 

FIG. 203 shows the same web page as that shown in FIGS. 202 generated on an actual 
800 by 600 pixel window by a standard Web browser. 

One can see that the size of images, column widths, and the horizontal displacement of 
both images and columns is smaller in FIG. 202 than in FIG. 201 . One can also see that although 
the size of the font is the same in both figures, the font is larger relative to the size of images, 
column widths, and the horizontal displacements of both images and columns in FIG. 202. 

The comparison of FIGS 201 and 202 shows that the 800 by 600 virtual layout mode 
allows a larger horizontal portion of a web page's layout to be displayed on a 320 by 240 screen at 
one time. The comparison of FIG. 203 to FIG. 202 and 201 show that the 800 by 600 virtual 
resolution view of FIG. 202 is much more similar to an actual 800 by 600 display of a web page 
than is the 640 by 480 virtual resolution view of FIG. 201 . 
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Thus, it can be seen that the use of the 800 by 600 virtual resolution view allows a user of 
a small screen computer to see a web page displayed in a manner that is more similar to that in 
which it would appears on the larger screen of a desktop or laptop computer. 

It should be appreciated that in other embodiments of the invention other, smaller or larger 
virtual screen resolutions could be used for either the first or second scale views. For example, it 
would be possible to layout a screen at a virtual resolution of 1024 by 768 pixels, and display at on 
the 320 by 240 screen shown in FIG. 193 by using a virtual resolution command.similar to the 
command 19110 shown in FIG. 191 that requested a virtual resolution of 1024 by 768, that 
requested the use of the same smaller font size, and that requested a scale factor of 3.2. 

It should also be appreciated that the present invention is not limited to use with display 
that are 320 by 240 pixels. For example, the innovation of decreasing the size at which a web 
page's images, column widths, and horizontal displacements are displayed while at the same time 
increasing relative size of fonts relative to such features to maintain the readability of text could be 
used on lower resolution screens. 

For example, way of scaling down a web page's layout could be used to map a larger 
portion of a web page into a cell phone screen. For example, by using the same 2.5 scale factor 
used in the 800 by 600 virtual view described above, a 440 virtual pixel width could be displayed on 
a cell phone having a 176 pixel wide screen. Other types of small computers, such as wrist watch 
computers, can also use the inventions scheme for scaling down a web page's layout. 

This innovations could be used on screens that have a higher resolution than 320 by 240. 
For example, it is currently possible to manufacture PDA sized screens that have a 640 by 480 
resolution. This innovation could allow one to select between an actual 640 by 480 resolution view 
on the screen and a scaled-down 800 by 600 or 1024 by 768 virtual resolution view on the same 
screen. The innovation can also be used to display scaled-down views of higher resolution layouts 
onto sub-portions of larger resolution screens, such as those used on desktop, laptop, and tablet 
computers. 
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In many embodiments of the invention the first, second, and third scale views each use a 
separate common scale factor to scale down the size of images, column widths, and horizontal 
displacements for images and columns, so that the layout produced at each view provides as 
faithful a representation of the web page as it would look as if actually display at the view's 
associated virtual resolution. In other embodiments, the scale factors associated with different 
layout elements could be different. For example, images might be scaled down more than column 
widths, or visa versa. 

In the embodiment of the invention described above the three scaled views are produced 
by telling a browser's layout engine that the web page is being laid out at the virtual layout 
resolution, and then scaling down the sizes and positions of the web page's elements for actual 
display. The invention is not limited to using this method. For example, in other embodiments the 
browser's layout engine could be modified to scale all references to pixel size, including the pixel 
size of images, column widths, and horizontal displacements, down by the scale factor before the 
layout, and then perform the layout at the actual pixel resolution at which the web page will be 
displayed. 

Many embodiments of the invention, like that described above with regard to FIGS. 191 
through 203, allow the user to select the set of one or more parameters that decrease the size at 
which web page elements such as.images, column widths, and horizontal displacements are 
displayed and the set of one or more parameters that increase the size at which fonts are 
displayed relative to such elements as a group, without requiring the user to separately select 
these different sets of parameters. This has the advantage of allowing a user to more quickly 
select between the different scale views. However, in some embodiments of the invention the user 
may be forced to separately select the set of one or more parameters that scale down images, 
column widths, and horizontal displacements, and the set of one or more parameters that control 
relative font sizes. 

It should be understood that the foregoing description and drawings are given merely to 
explain and illustrate, and that the invention is not limited thereto except insofar as the interpretation 
of the appended claims are so limited. Those skilled in the art who have the disclosure before them 
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will be able to make modifications and variations therein without departing from the scope of the 
invention. 

The invention of the present application, as broadly claimed, is not limited to use with any 
one type of operating system, computer hardware, or computer network, and, thus, other 
embodiments of the invention could use differing software and hardware systems. 

Furthermore, it should be understood that the program behaviors described in the claims 
below, like virtually all program behaviors, can be performed by many different programming and 
data structures, using substantially different organization and sequencing. This is because 
programming is an extremely flexible art in which a given idea of any complexity, once understood 
by those skilled in the art, can be manifested in a virtually unlimited number of ways. Thus, the 
claims are not meant to be limited to the exact steps and/or sequence of steps described in the 
figures. This is particularly true since the pseudo-code described in the text above has been highly 
simplified to let it more efficiently communicate that which one skilled in the art needs to know to 
implement the invention without burdening him or her with unnecessary details. In the interest of 
such simplification, the structure of the pseudo-code described above often differs significantly 
from the structure of the actual code that a skilled programmer would use when implementing the 
invention. Furthermore, many of the programmed behaviors that are shown being performed in 
software in the specification could be performed in hardware in other embodiments. 

In the many embodiment of the invention discussed above, various aspects of the 
invention are shown occurring together that could occur separately in other embodiments of those 
aspects of the invention. 

Most of the various illustrations of subpixel optimization and non-linear color-balancing 
described in various parts of this specification relate to RGB subpixel addressable displays having 
vertical subpixel striping. It should be appreciated that many of the innovations relating to non- 
linear color balancing and subpixel optimization can be used with subpixel displays that have BGR 
or other types of subpixel addressability, as well as subpixel displays having horizontal subpixel 
striping. 
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In the non-linear color balancing methods shown above the only portion of a subpixel's 
luminosity distributed by color balancing is that portion that is higher than the minimum subpixel 
luminosity value within a pixel. But in other embodiments other portion of a subpixel's luminosity 
that cause color imbalance within a pixel could be distributed, such as portions that differ from the 
mean or maximum subpixel luminosity of pixel. In such embodiments subpixel luminosity values 
below such a mean or maximum would, in effect, be negative luminosity values, that could be 
distributed by a weighted decreasing of subpixel luminosities in such a subpixel's neighborhood. 

All the non-linear color balancing methods shown above only distribute those portions of a 
subpixel's luminosity that cause color imbalance within a subpixel's corresponding pixel. This is 
done because the arrangement of three successive RGB or BGR subpixels commonly found within 
a whole pixel are perceptually well color balanced. If the subpixels of such a whole pixels are of 
equal luminosity they tend to appear more color balanced to the eye than an isolated set of the 
same three colored subpixels shown at the same intensity in an order in which green is not the 
central color. This is one of the reason why edges of fonts that appear at other than whole pixel 
boundaries appear color imbalanced. 

But other non-linear color balancing embodiments need not be limited to only distributing 
subpixel luminance that causes imbalance within individual whole pixels. Other non-linear color 
balancing embodiments could determine the degree of subpixel color imbalance within regions 
other than whole pixels, and distribute subpixel luminance values based totally or in part on 
imbalance with such regions. For example, studies could be performed to find which distributions 
of imbalanced coverage values created a minimal spatial spreading while maintaining the 
perception of color balance, for each of a plurality of commonly occurring imbalance patterns, and 
such perceptually selected distributions could be used to distribute color imbalance that occurs in 
spatial regions other than whole pixel regions. 
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